1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Reliable transfer of .TS files?

Discussion in 'TiVo Home Media Features & TiVoToGo' started by alleybj, Feb 23, 2016.

  1. May 6, 2017 #81 of 141
    wuzznuubi

    wuzznuubi Member

    130
    4
    Jan 16, 2013
    @Dan203
    If you still have MPEG2 recordings try one of those using both PS and TS downloads and compare the result. You may be able to figure out what is causing the glitch. I really think it's something in the TiVo units code that packs it as a TS TTG where maybe a glitch, dropout, timestamp issue, counter wrap or something messes them up. It was introduced aroung Feb., 2016 in both my Premiere and Roamio downloads using TS. I'll see if I can find the ouput from TiVoLibre on the last couple TiVo TS downloads I robodownloaded. It was always hit-or-miss. Some recordings downloaded without issues first time, others took many (5-35) retries downloading back-back to get one without transport stream sync loss (I know you know that means loss of TS packets/info like cutting out part of a program you're streaming and you can't rewind the stream, or a broadcast, so no way to recover what isn't there).
     
  2. May 6, 2017 #82 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    I'm looking at the error. Multiple downloads have slightly different data in the damaged areas, but the damage always starts at the same byte. The damage is in the .tivo file, so it's not the decrypting causing the error.

    Does anyone know if the TiVo supports partial downloads with an offset byte? I'm wondering if I'll get the error to move if I start from a different point.
     
  3. May 6, 2017 #83 of 141
    wuzznuubi

    wuzznuubi Member

    130
    4
    Jan 16, 2013
    I think that it's only supported for PS downloads and not sure if or when that may have been deprecated if it was. Last time I used that function was too long ago to remember.
     
  4. May 7, 2017 #84 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    Is there anything else that causes the error to move or stop? Like putting the tivo into standby? Or setting all the tuners to a channel you don't get?
     
  5. May 7, 2017 #85 of 141
    wuzznuubi

    wuzznuubi Member

    130
    4
    Jan 16, 2013
    Not that I have found. I will upload my results of a sixteen out of twenty download attempt with one recording where the TS Sync loss is all over the place and the sixteenth download produced a .TiVo TS file w/o sync loss. There is definitely a moth in the code that TiVo uses to give us a TTG TS download vs. their old PS download method.
     
  6. May 7, 2017 #86 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    Are 1-15 all different? Or do most/all of then have the error in the same place?
     
  7. May 7, 2017 #87 of 141
    ej42137

    ej42137 Well-Known Member

    2,329
    38
    Feb 15, 2014
    Los Angeles
    You can transfer TiVo to TiVo starting from a paused point.
     
  8. May 7, 2017 #88 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    I know that. I meant TiVo to PC. I wanted to see if I started the transfer in a different spot if the error section would change. I messed around with the HTTP headers but it didn't work. It still downloaded the entire file.
     
  9. May 7, 2017 #89 of 141
    gonzotek

    gonzotek tivo_xml developer

    2,421
    9
    Sep 24, 2004
    Outside...
    kmttg has code to transfer from paused point, here's the wiki on the topic:
    kmttg / Wiki / Resume_Downloads
    Note, it says it's not available for TS transfers near the bottom of the page. But last time I tried it (I believe it was sometime last year), the process worked for an mpeg2 ps transfer. Not sure if that's helpful to the topic, but it's all I got :)
     
  10. May 7, 2017 #90 of 141
    wuzznuubi

    wuzznuubi Member

    130
    4
    Jan 16, 2013
    Below is the output of my script last time I ran it that runs all 20 tivo ts downloads through tivo-libre looking for TS sync loss. Using grep or find I look for the phrase "Re-synched" in the output of tivo-libre and if it doesn't occur for that tivo file, then no TS sync loss. In this case it was the sixteenth download, filename TivoTSdownload_16.TiVo
    Code:
    TivoTSdownload_01.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    TivoTSdownload_02.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    Re-synched at packet 6091519 (byte 0x444356ec)
    TivoTSdownload_03.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    Re-synched at packet 4568606 (byte 0x333293bc)
    Re-synched at packet 7614637 (byte 0x5554f5b0)
    TivoTSdownload_04.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    Re-synched at packet 6091519 (byte 0x444356ec)
    TivoTSdownload_05.TiVo
    Re-synched at packet 3045398 (byte 0x222047a8)
    TivoTSdownload_06.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    Re-synched at packet 3045253 (byte 0x222047a8)
    TivoTSdownload_07.TiVo
    Re-synched at packet 3045398 (byte 0x22202eb4)
    Re-synched at packet 3045433 (byte 0x22204864)
    Re-synched at packet 7614930 (byte 0x5554f5b0)
    TivoTSdownload_08.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    TivoTSdownload_09.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    Re-synched at packet 3045253 (byte 0x222047a8)
    TivoTSdownload_10.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    Re-synched at packet 3045253 (byte 0x222047a8)
    Re-synched at packet 6091488 (byte 0x444356ec)
    TivoTSdownload_11.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    TivoTSdownload_12.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    Re-synched at packet 3045253 (byte 0x222047a8)
    Re-synched at packet 4568573 (byte 0x333293bc)
    Re-synched at packet 7614604 (byte 0x5554f5b0)
    TivoTSdownload_13.TiVo
    Re-synched at packet 6091666 (byte 0x444356ec)
    TivoTSdownload_14.TiVo
    Re-synched at packet 7614929 (byte 0x5554f5b0)
    TivoTSdownload_15.TiVo
    Re-synched at packet 3045398 (byte 0x222047a8)
    Re-synched at packet 4568718 (byte 0x333293bc)
    Re-synched at packet 6091486 (byte 0x444356ec)
    Re-synched at packet 7614716 (byte 0x5554f5b0)
    TivoTSdownload_16.TiVo
    TivoTSdownload_17.TiVo
    Re-synched at packet 4568751 (byte 0x333293bc)
    TivoTSdownload_18.TiVo
    Re-synched at packet 3045398 (byte 0x222047a8)
    Re-synched at packet 6091633 (byte 0x444356ec)
    TivoTSdownload_19.TiVo
    Re-synched at packet 1521712 (byte 0x110dad00)
    TivoTSdownload_20.TiVo
    Re-synched at packet 3045398 (byte 0x22202eb4)
    Re-synched at packet 3045433 (byte 0x22204864)
    Re-synched at packet 4568752 (byte 0x33326cd0)
    Re-synched at packet 4568758 (byte 0x333272b0)
    Re-synched at packet 4568803 (byte 0x333293bc)
    
     
    ClearToLand likes this.
  11. May 7, 2017 #91 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    Congratulations on your FIRST LIKE! :clapping: :D

    I appreciate all the help / information you've been giving / sharing with @Dan203 in the pursuit of solving this long-standing TiVo TS / 'Fast' Format transfer BUG. :thumbsup:
     
  12. May 7, 2017 #92 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    I could actually make this a feature of pyTivo. I could scan the bytes as they're downloaded looking for ts sync loss. Even without decrypting. TS files are pretty simple. They should have a byte 0x47 every 188 bytes. If you skip ahead 188 bytes and it's not 0x47 then you've lost sync. If it's lost then I could stop the transfer immediately and restart it. Not sure if it would significantly slow down downloads though. I'd have to try it.
     
  13. May 8, 2017 #93 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    Well, one thing for sure is folks haven't been pleased with being 'surprised' days / weeks / months later that their .TIVO TS / 'Fast' Format transferred files are corrupt (and have been since Day 1) and now they have to 'mess around' with decrypting them into .TS files (or buy VideoReDo) in order to view them.

    Regarding the potential slowdown, some ideas immediately come to mind.

    Make the whole process optional (checkbox on entry line in NPL):
    1. Option 1 (unchecked) - operate the 'old' way (ignore the errors; sounds dumb though...)
    2. Option 2 (checked) - use the new 'Check for TS Sync Loss' routines
      • Enter a number, 01-99, for maximum retries
      • If maximum retries reached WITH success
        - place the # of retries at the end of the field in NPL
        (useful to gauge how much extra time / effort it is consuming)
      • If maximum retries reached WITHOUT success
        - 'highlight' entry somehow in NPL (pick a character to place at the end of the field: !, @, #, etc...)
        - post error to log
    Just 'thinking out loud...'
     
  14. May 8, 2017 #94 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    @Dan203,

    BTW, has CNTL-CLICK/SPACE (individual) and / or SHFT-CLICK/SPACE (range) for selecting NPL entries to transfer been considered / implemented?
     
  15. May 8, 2017 #95 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    @Dan203

    Have you begun creating formal 'Release Notes' yet?
     
  16. May 8, 2017 #96 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    I've been thinking about adding some sort of transfer log to the desktop app, so I could just add the retry/sucess/fail stuff to that. The best part about doing it this way would be that I could kill the transfer as soon as a discontinuity was detected, so I could potentially save some time compared to downloading the full file and scanning with tivilibre.

    I'm considering it, but that list I'm using doesn't support that functionality natively so it would require some extra work.
     
  17. May 8, 2017 #97 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    There are release notes on the website. That's really all I've got.
     
  18. May 8, 2017 #98 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    Aha!

    I never CLICK'd on 'Download' because I wasn't ready to d/l it yet so all I saw was the moving images.
    • Have you considered separating the 'Release Notes' from the 'Download' button?
      .
    • Do you have 'Release Notes' prior to 1.5.9?
      .
    • Is a Wiki 'How-To' set of pages being considered?
     
  19. May 8, 2017 #99 of 141
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    523
    46
    Jul 9, 2001
    Central Jersey
    Checking for 0x47 every 188 bytes is definitely the way to go - that's why I suggested adding a variable, user-selected 01-99 maximum retry count.

    Now, getting all of this 'juicy stuff' onto the NPL is another matter...
     
  20. May 8, 2017 #100 of 141
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,252
    838
    Apr 17, 2000
    Nevada
    No I didn't start keeping track of changes until 1.5.9. But the vast majority of changes were bug fixes prior to that, so there wasn't much to report anyway. 1.6.1 is pretty stable. (I hope :)) But I'm still engrossed in the project, so I've still got a lot of ideas for new features and improvements.

    Although I'm thinking about trying to recruit some beta testers so I don't keep having these releases where I discover some bug the next day and have to annoy my users again with the update message.
     

Share This Page