Java port of TivoDecode

Discussion in 'Developers Corner' started by fflewddur, Sep 6, 2015.

  1. Nov 1, 2015 #101 of 154
    mackworth

    mackworth cTiVo Developer

    97
    3
    Jul 31, 2006
    Well, Java certainly makes the binary more portable, but it also adds a requirement to have the Java VM installed. My impression from the thread was that tivodecode-ng was a pretty complete implementation of the transport stream problem, but that there were a few intransigent bugs, which fflewddur manage to figure out and fix for tivolibre. I was hoping those changes could get ported back.
     
  2. Nov 8, 2015 #102 of 154
    dougblair

    dougblair New Member

    5
    0
    Feb 22, 2009
    Hello...

    I have been a kmttg user for several years and have just discovered this thread. AMAZING work, all of you! It's a true delight to watch is kind of collaborative development work across several months, from different locations, and maybe some of you have real jobs :). Your generosity with your time and skills and expertise is greatly appreciated.

    Please know that if I ever get away from my own TiVo downloading, decoding, transcoding and uploading to AWS addictive activities (resulting in some 60+ TB of old TV shows, now uploading to the Amazon Unlimited storage cloud, which is setting some kind of consumer account record) and find myself with enough time to actually watch some of the programs, I will be thinking about all your hard work!

    Thank you!
    Virtual beers! (Other suggestions will be accommodated!)

    Doug Blair, Chicago
     
  3. herbman

    herbman Member

    120
    0
    Apr 8, 2008
    This may be old news, but I've been successful with the following pattern from a bash/cygwin shell:

    Code:
    wget -O - 'http://tivo:<mak>@....' | java -jar tivo-libre-0.7.2.jar -m <mak> | ffplay -
    Thank you for this convenient way to watch items directly without steps!

    2 tuner premiere on ethernet.
     
  4. herbman

    herbman Member

    120
    0
    Apr 8, 2008
    Any chance you have a unit test which runs this a number of times to benchmark? I'm happy to play with the code but I'd love to have a straightfoward way to know if there's an impact.
     
  5. fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Nice! TivoLibre also caches your MAK, so if you're scripting this, you shouldn't even need the -m parameter after the first run.
     
  6. fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    I don't; I use VisualVM to profile Java, so I manually run TivoLibre on a large file while capturing CPU and memory usage in VisualVM. The exact CPU time varies too much to be useful, but the percentage of time spent in each method is very stable across invocations, so I just pay attention to that. If you're optimizing the Turing code, you should see the percentage of time spent in the I/O methods increase while the time spent in the generateTuring() method decreases.
     
  7. forumuser

    forumuser New Member

    5
    0
    Mar 3, 2014
    I have several recordings on my tivo (premiere) which fail to fully download when retrieving in PS format, but download fully when downloaded in TS format.

    However, the partial PS download (after decoding) plays back without hickups, but the TS download (after decoding) plays back with a ton of issues, even during the same spots where the PS file plays normally. The issues are primarily on the video: lots of artifacts.

    The above occurs with both tivodecode (0.4.4) and tivolibre (latest).

    Is this normal/expected behaviour? Analyzing this from a blackbox perspective, it would seem that the tivo's ts_to_ps converter on the device itself is able to fix issues with the TS file, but chokes when handling some of those issues (this is what I presume aborts the transfer of the PS formatted file). So, I am wondering if something similar could be done on the external decoding software (eg. tivolibre), but without the choking! :)
     
    Last edited: Jan 12, 2016
  8. moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    You should check how TiVo DirectShow Dump decrypts it since that is THE standard to compare against. If it has issues as well then the problem is in the .TiVo file itself.
     
  9. forumuser

    forumuser New Member

    5
    0
    Mar 3, 2014
    The same problem occurs when I use the "directshow dump" utility on windows.

    What is curious to me is that, as I understand it: Tivo stores the recordings in its harddrive in the the native format, which is "MPEG-2 transport stream" (aka TS). Note that MPEG-2 here has nothing to do with the video codec used for the video elementary stream within.

    So, if one pulls the recording off the unit requesting it in TS format, then one is getting a copy of what is on the tivo's harddrive.

    However, if one pulls the recording off the unit requesting it in PS format, then the tivo has to take what it has in its harddrive and convert it from TS to MPEG-2 PS format. Again, MPEG-2 here is not related to the elementary video stream within.

    If the above is correct, then why is it that the "processed" file (i.e. the PS file) after being decoded plays back normally, while the source file (ie. TS file) after being decoded does not?

    It has been almost a decade since I worked on MPEG-2 streams, but I think this has something to do with filler packets needed to maintain a bit rate or a clock or something like that. I can't recall right now.

    Anyhow, I think we should be able to fix this in the post processing software (eg. tivolibre), just as the tivo's software does when it converts from TS to PS on the fly during the transfer.

    I will dig in a bit into these streams.
     
    Last edited: Jan 12, 2016
  10. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    51,001
    6,044
    Apr 17, 2000
    Nevada
    The TiVo does not store the file as a TS. They store the audio and video data separately as chunks and then links the chunks using a database. So regardless of which format you download the audio and video are being remuxed and encrypted on the fly. TS is a simpler format so it allows them to have better tollerance for stream errors, but the encryption process can still cause issues with corrupt streams.
     
    ClearToLand likes this.
  11. forumuser

    forumuser New Member

    5
    0
    Mar 3, 2014
    Thanks Dan203. That explains a lot.

    Are there any known reasons as to why Tivo stores the recordings as you describe as opposed to using the source format?

    It seems strange to do that, but I am sure they had good reasons to do it... I wonder what they gain by doing that. Anyone know? Just curious...
     
  12. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    51,001
    6,044
    Apr 17, 2000
    Nevada
    The TiVo code started it's life in the SD world where they had to actually transcode analog audio/video to store it on the hard drive. I assume the way they store the streams was designed around that. When the S3 units came along and they started recording the digital streams directly it was probably easier to demux the incoming transport streams and store them the old way then it was to redesign the whole system to work around storing the transport streams directly.
     
  13. forumuser

    forumuser New Member

    5
    0
    Mar 3, 2014
    Cool. Thx for that insight. I forgot that this product has been around since before the HDTV changeover! :)
     
  14. global_dev

    global_dev Member

    154
    1
    Mar 15, 2010
    Northern VA
    great work guys! having some problems though



    i just tried to change from tivodecode to tivolibre and i am getting some weird behavior, the log shows that it starts a job doenloading and decrypting to .mpg, but then it changes on the screen to .ts and i don't see the .mpg again (even in the log). perhaps i am doing something inappropriate. i've attached some screenshots and the log, maybe you guys can help.

    I had no problems with the tivodecode + decrpyt prior. just looking to speed up the jobs.

    Premiere
    OSX El Cap
    kmttg 2.1h ( i know the pics indicate an older version, but that's just the download spot w/upgrades)
    tvlibre 0.7.2.

    [​IMG]

    [​IMG]

    [​IMG]

    log

    Code:
    
    >> DOWNLOADING/DECRYPTING TO /Users/j/TEST_VIDEO__2007.ts ...
    Configuration saved to file: /Users/j/kmttg/config.ini
    
    Refreshing encoding profiles
    TEST_VIDEO__2007.ts: size=1682.38 MB elapsed=0:04:27 (52.86 Mbps)
    
    ---DONE--- 
    
    job=tdownload_decrypt output=TEST_VIDEO__2007.ts
    
    >> Running ccextractor on TEST_VIDEO__2007.ts ...
    "ccextractor.0.63/ccextractor 0.6.3" "TEST_VIDEO__2007.ts" 
    
    ccextractor job completed: 0:00:26
    
    ---DONE--- 
    
    job=captions output=TEST_VIDEO__2007.srt

    couldn't embed the vid, so the link is below of the behavior

    https://youtu.be/uZI3BPZcN1I
     
    Last edited: Jan 26, 2016
  15. global_dev

    global_dev Member

    154
    1
    Mar 15, 2010
    Northern VA
    so if i unmark download via TS, it all seems to work.. any reason i shouldn't try to get TS downloads working other than PS seems to work?
     
  16. moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    global_dev, 2 things:
    1. There is no need to replace tivodecode with tivolibre as you are doing. tivolibre jar is embedded in kmttg so as long as you turn on option in config as you did that's all you need.

    2. When you switch to TS downloads the suffix changes from .mpg to .ts by design because .mpg implies mpeg2 program stream container, and .ts implies transport stream container. So I don't see any issues at all based on what you are showing.
     
  17. moelleref

    moelleref New Member

    7
    0
    Jan 27, 2016
    I'm using kmttg v2.1h and I've noticed that after downloading a number of files from my tivo and decrypting them into the mpeg ts format, they are highly pixilated. So much so they are unwatchable. I do not see any pixilation when I watch the recording on my tivo.

    Currently it seems to be happening mostly with recordings from FOX HD.

    I have a Tivo Premier XL, and I'm downloading the files in transport stream format. I'm using tivolibre to decrypt the file.

    I've also re-downloaded the videos in PS format and decrypted them using tivolibre and they look ok.

    A sample of the tivo file and the ts file can be found at the following location.

    drive.google.com/folderview?id=0B1HsrgAVb3dxY0k3akdReEdBekk&usp=sharing


    Any ideas what my be causing this issue when decrypting them into a ts file?

    Thanks for any help.
     
  18. moyekj

    moyekj Well-Known Member

    12,157
    809
    Jan 23, 2006
    Mission...
    fflewddur, I can confirm the above sample from moelleref seem to be a problem with tivolibre and TS decrypt. Decrypting with DSD results in a video that plays fine. (I obtained moelleref's 10 digit MAK via PM/email). Note that moelleref also confirmed that downloading in PS container and decrypting that way works fine, so it's TS specific decrypt issue.
     
  19. fflewddur

    fflewddur R&D

    169
    3
    Jul 20, 2015
    Seattle, WA
    Cool, thanks for the looking into this! Moelleref, can you PM me your MAK so I can investigate this myself and figure out where TivoLibre is going wrong?
     
  20. moelleref

    moelleref New Member

    7
    0
    Jan 27, 2016
    fflewddur, I do not have enough posts yet to send PMs. However, if you PM me your email address (I can read PMs; however, I can't reply), I'll email my MAK to you.
     

Share This Page