1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Reliable transfer of .TS files?

Discussion in 'TiVo Home Media Features & TiVoToGo' started by alleybj, Feb 23, 2016.

  1. May 9, 2017 #101 of 153
    reneg

    reneg Member

    763
    21
    Jun 19, 2002
    I've been playing with a Windows batch script which I plan to make available after the next kmttg update which is similar to the robo-download script that wuzznuubi posted about earlier in this thread. I've been testing my batch script on two Tivos; a 4 tuner Bolt & a 4 tuner Roamio. Both are connected OTA to the same amplified antenna with a splitter between the two Tivos. On the networking side, both Tivos are connected to the same switch which also connects my PC.

    What I am seeing in my transfers is that I cannot get a clean download from the Bolt. After 180+ downloads of several different tivo files, I have yet to produce a clean transfer. I can get clean downloads from the Roamio, and often on the first attempt. The Bolt is transferring data about three times faster than the Roamio, I've tried several things to change the dynamics of the downloads with no effect.

    I don't believe there is a known method of resuming TS downloads at this time.

    With the results (or lack therefore of) that I've seen on my Bolt, I believe a faster method to a clean TS download may be to stitch together parts of two or more downloads to form a clean download. This method is not guaranteed to yield a clean download if the same packet fails to re-sync every transfer.

    I've uploaded the download data for both my Tivos if someone wants to look at it.
     

    Attached Files:

    ClearToLand likes this.
  2. May 9, 2017 #102 of 153
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    606
    50
    Jul 9, 2001
    Central Jersey
    I'm in the midst of assembling another chrome storage rack so I just found your new (8 minutes old) update during a 'rest break'. The Bolt is 1000Mbps while the Roamio is 100Mbps (wired).

    Two thoughts on the Bolt problem come to mind:
    1. If you have an old 100Mbps switch, or router, laying around, run the Bolt through that to throttle its speed.
      .
    2. I've just gotten into 'Managed' switches since Newegg was practically giving them away at approximately the same price as 'Unmanaged' switches.
      - Let me go upstairs and glance at the manual (in PDF) and / or log onto one of the several in service on my LAN and see if 'throttling' is an option.
      - I know that QoS is since I have set QoS (lowest-to-highest priority) as:
      1. Default
      2. TiVo-to-TiVo and to kmttg / pyTiVo / Streambaby PC
      3. Switch-to-Switch (i.e. UPLINKS)
      4. Main Switch-to-Router (only *ONE* LAN port is in use on the router)
     
  3. May 9, 2017 #103 of 153
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    I was thinking about throwing together a program that would take 2+ .tivo recordings and try to stitch them together into a clean recording. But in my tests the files that fail always have the failures in the same place, so that wouldn't really work.

    I'm going to try to add the sync scan to pyTivo soon, so we can try to replicate the robo download you're doing but perhaps with a little bit of saved time since I can stop the download as soon as I detect an error.
     
    ClearToLand likes this.
  4. reneg

    reneg Member

    763
    21
    Jun 19, 2002
    I got rid of all my old, slow switches when I moved. I changed where I ran the batch script from on the PC side. For my previous tests, I was running the batch file from an SSD on my PC. I switched to running the batch file from a hard drive and low and behold, I got a clean transfer from my Bolt on the very first try. The transfer was still about twice as fast as the Roamio. I'm going to retest the transfers that failed previously to see if this was an anomaly.
     
  5. lpwcomp

    lpwcomp Well-Known Member

    8,813
    130
    May 6, 2002
    John's...
    I'm seeing better TS transfers since I started idling everything as much as possible. No active recording on the TiVo and PC as idle as possible, especially making sure that nothing except kmttg is communicating with the TiVo from the PC.

    I still get occasional glitches but they are less "intense".

    I use VRD for everything except the actual transfer
     
  6. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    Does throttling the download help at all? I could probably put some artificial throttling into pyTivo if it would actually help.
     
  7. reneg

    reneg Member

    763
    21
    Jun 19, 2002
    My successful transfer may have been an anomaly. I have not had another successful transfer from my Bolt in the last six hours. Nothing else going on with that Bolt at all. I'm not convinced throttling would help.
     
  8. lpwcomp

    lpwcomp Well-Known Member

    8,813
    130
    May 6, 2002
    John's...
    I haven't tried that but I doubt that it would help. I guess it's possible, but I believe it has been tested by others and had no effect.

    My theory is that at least part of the problem is that the TiVo TS transfer code doesn't handle being interrupted very well and sometimes produces an incomplete block in the transfer buffer, but that is just a (S?)WAG.

    Unfortunately, there is no way to force the TiVo to dedicate itself to the transfer, plus it seems likely that there is another bug whereby it has problems with some sequences, so that some glitches always appear at the same spot.
     
  9. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    Holy sh*t I think I just figured out how to fix this. I added code to pyTivo to download just the header first, then I setup the remaining loop to download the bytes in 188 byte aligned chunks and I got a clean download of a file that always has an error in the exact same spot.

    Edit: False alarm. Second download produced an error at the same place, so apparently the clean one was just a fluke. :(

    Edit: 3rd attempt was clean again. This file consistently was not clean before, so this is an improvement. I'm going to keep playing with the buffer sizes and see if I can make it consistent.
     
    Last edited: May 10, 2017
    mlippert and ClearToLand like this.
  10. ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    606
    50
    Jul 9, 2001
    Central Jersey
    Rah! Rah! Sis Boom Bah!
    "0x47 @ 188"
    (Cheer)
    :clapping: Go Dan Go! :clapping:
    Touchdown! :handok:
     
    gonzotek likes this.
  11. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    I've got the retry thing working. The downloads are just completely sporadic though. Even when I get past the point where the first error is in my test file there is one near the end that kills it. I've only gotten two clean downloads in dozens of attempts.
     
    gonzotek and ClearToLand like this.
  12. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    I've got an idea for how to handle this that might be cool....

    The simplest option will be to simply halt the download and start a new one as soon as a sync loss is detected. But that could result in dozens of attempts with no clean downloads. So my idea is to keep track of how many bytes are effected by the sync loss and try to keep the best copy of the show. This means that the first attempt will always download the full show. After that subsequent attempts will compare their number of effected bytes to that file. As soon as they match or exceed the number of effected bytes the download will abort. If they reach the end with less bytes effected, but still have errors, then the original file will be deleted and this will become the new benchmark that subsequent attempts are compared against. At the end of X iterations (set by user) you will be left with a complete download with the least number of bytes effected across all attempts. At the end I would rename the file so that the file name contained the number of bytes effected by sync loss.

    Sound good?
     
    mlippert likes this.
  13. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    OK I got this working. I'm going to have 3 options...

    1) Ignore. Which will download the file once, but will still mark the file name with the number of effected packets
    2) Best file. Which will work as outlined above retrying X times and keeping the file with the least number of errors
    3) Fail. Which will retry X times but will abort the transfer as soon as it hits and error and if after X tries there isn't a clean copy it will leave nothing on disk and only a fail message in the queue.

    In my tests some files could never be downloaded completely clean, but I could get versions with less errors then the initial download.
     
    mlippert likes this.
  14. ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    606
    50
    Jul 9, 2001
    Central Jersey
    Wow Dan! :eek:

    I didn't want to monopolize this 'conversation', but it's been almost NINE hours and *NO ONE* else has replied to your questions?!? (I wanted to give other folks a chance to participate too.)

    That is DEPRESSING, when you're hot into programming a solution to a problem, ask the 'Community' for some feedback and *NO ONE* replies - I apologize; I thought that I was giving the other forum members 'space'. I'm so sorry...

    #2 :thumbsup:
    #2 again, but with a 'twist'. ;)

    OK. My original idea was based on using tivolibre, finding an error and then retrying 01-99 times based on the user selection PER SHOW on the NPL. You, @reneg, and @wuzznuubi have subsequently displayed that some shows will *NEVER* TS / 'Fast' Format transfer cleanly. So, what do we do now?

    I appreciate your implementing the "0x47 @ 188" logic. Today (well, now it was yesterday), I read a post on the AVS Forum (in either the Hauppauge PVR-1212 thread or the PVR-1512 thread) where a fellow who couldn't tolerate *ANY* 'glitches' (in copies from his DVR to his PC) used VideoReDo to 'Cut-N-Paste' segments from MULTIPLE downloads to create one final 'clean' copy. Now, I certainly don't expect you to do that - I'm just presenting it as one person's solution to the 'glitch' problem.

    Based on the facts that I currently have available to me, I vote for #2 "Best File" *WITH* a possible addendum. AFAIK, since this TS / 'Fast' Format transferred "Best File" *STILL* has 'glitches', neither pyTiVo (all versions) or Streambaby will be able to transfer / stream it back to a TiVo for viewing; i.e. it will need to be run through either VideoReDo or tivolibre, right? So, how about following up any unsuccessful TS / 'Fast' Format transfer with a PS / 'Slow' Format transfer (append "_PS" to the filename)? At the very least, we can be fairly confident that it will be viewable (transfer or stream).

    Also, I'd like to see the "Total # of Retries Used" on the NPL on the detail line next to the "Total # of Retries Requested" and, for EACH retry, I'd like to see a "# of Sync Errors Detected" per retry attempt in a log file.

    And this brings up a question that I intended to post in a separate thread:

    What are the PROs and CONs of TS vs PS Format transfers?

    Back in the summer of 2016, I was *EXTREMELY* disappointed to discover that ~66% of the TS / 'Fast' Format transfers that I made (per the TCF recommendations) via kmttg and subsequently attempted to view via pyTiVo *FAILED*! :mad: I was a Newbie - I'd didn't know better - I read LOTs of TCF posts; I read the kmttg and pyTiVo Wikis. And yet, I still got 'screwed'. :rolleyes:

    AFAIK, PS / 'Slow' Format transfers have the tendency to lose Closed Captions - are there any other known CONs?

    BOTTOM LINE: Since we can be almost 100% confident that a PS / 'Slow' Format transfer TiVo-to-PC will be able to successfully transfer / stream PC-to-TiVo, I see no downside to using that as a 'fallback' to a TS / 'Fast' Format transfer failure.

    Others are welcome to join in and comment.
     
  15. Mikeguy

    Mikeguy Well-Known Member

    4,346
    638
    Jul 28, 2005
    That raises an interesting possibility--an option to fall back to a PS transfer if the TS doesn't work successfully. But then, I guess the question is, what would that point be (how many errors would that be/how big are those errors) and is it possible to set such a point?

    I'm seeing that the PS issues are transfer speed and possible closed caption loss--can something be done on the CC side to make PS a failsafe alternative/fallback? Perhaps a similar, try-again approach?
     
  16. gonzotek

    gonzotek tivo_xml developer

    2,429
    9
    Sep 24, 2004
    Outside...
    PS transfers preclude h.264 video, which is becoming more and more prevalent in the field (Comcast is converting to it nationwide, excluding only locals, for now). For me that's the biggest con of a PS transfer - I can't even use it on a large portion of the channels I'd want to.
    Some older threads on the topic (mostly rehashing things already discussed):
    Difference between MPGEG-PS and MPEG-TS files?
    Downloading in PS vs TS Format: Pros and Cons
     
    ClearToLand and Mikeguy like this.
  17. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    I was thinking about adding codec detection and automatic switch to PS for MPEG-2 as an option. For that I'd need to find and read the PMT packet, since there doesn't seem to be any indicator in any of the XML data or the header that denotes the video format.
     
  18. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    FYI my last release, 1.6.1, included a feature to get past these errors. You'll still have a glitch where the error is but it wont completely fail to transfer.
     
    ClearToLand likes this.
  19. ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    606
    50
    Jul 9, 2001
    Central Jersey
    To be clear, the *ORIGINAL* TiVo-to-PC TS / 'Fast' Format transfer would have had to be performed with pyTiVo Desktop v1.6.1 in order for the PC-to-TiVo 'glitch-free' transfer back to occur, right?

    .TIVO TS / 'Fast' Format files transferred TiVo-to-PC via kmttg from early 2016 are not going to now 'magically' transfer PC-to-TiVo 'glitch-free' with pyTiVo Desktop v1.6.1?
     
  20. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    39,853
    1,034
    Apr 17, 2000
    Nevada
    They will not be "glitch free", but they will not just fail like they did before. I use tivolibre to skip over the bad parts so the file always completes the transfer. However there will be glitches where the bad parts are. How bad they are depends on how many packets were affected and what types of frames they encompassed. VideoReDo has better logic to repair these kinds of issues, so using it will produce better results, but with pyTivo Desktop 1.6.1 you can at least transfer your .tivo files without having to decrypt them first.

    I should have the new version with the code to retry downloads available in a couple days.
     

Share This Page