1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Reliable transfer of .TS files?

Discussion in 'TiVo Home Media Features & TiVoToGo' started by alleybj, Feb 23, 2016.

  1. ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    685
    60
    Jul 9, 2001
    Central Jersey
    Dan,

    I trust that you already know how highly I think of you - I've applauded your efforts to (at least attempt to) solve *EVERY* 'known-to-man ' TiVo transfer problem (TiVo-to-PC and PC-to-TiVo) with pyTiVo Desktop *MANY* times via either LIKEs, posts or PMs. :clapping: (I pushed VERY hard to you @Dan203 , Kevin @moyekj , and William @wmcbrine via PMs.)

    But... you need to learn how to answer a DIRECT question with a DIRECT answer. :oops:

    As an 'Old Fart / Baby Boomer / Grandfather / yada, yada, yada ', I've learned, through MUCH experience, to avoid being vague / stating an ABSOLUTE when it could really be a MAYBE. Remember, *EVERYTHING* entered onto the internet is *FOREVER* - there are no DELETEs; no changing your answer in a future post; whatever you say RIGHT NOW is *FACT* / "Cast-in-Stone". :rolleyes: ;)

    OK. Lightening up... pyTiVo Desktop v1.6.1. is no 'magic elixir ' - whatever shows that were previously transferred TiVo-to-PC in TS / 'Fast' Format via 'whatever ' tool other than pyTiVo Desktop v1.6.1 will *NOT* now be able to be transferred back PC-to-TiVo 'glitch-free ' with pyTiVo Desktop v1.6.1, or any other tool. (Let's use @oldradio99's grandchildren's shows as an example - sorry folks, you'll have to SEARCH for that thread on your own - too late / too tired).

    IANAL, *BUT*, IMHO you HAVE to be more explicit in your statements. Otherwise 'someone ' is most certainly going to misinterpret 'what you said from what you meant to say '. :) :kissingheart:
     
  2. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    No. There is no way to magically fix the errors that were introduced into the original download by TiVo. VideoReDo can do a bit more then scanning for the missing TS packet headers though. It can look for audio and video frames in the garbage data and reconstruct them, to an extent, which is why when you fix one of these files with VideoReDo the glitch is usually less severe then what you get with tivolibre, which simply dumps all the bad packets.

    As to what I did in 1.6.1... prior to that version if you uploaded a file with a glitch there was a good chance the TiVo would simply stall at the glitch and never even receive the rest of the file. By skipping over those bad packets the TiVo is able to continue downloading the remainder of the file and you end up with something that is at least watchable. In my tests thw glitch was usually only about 2-3 seconda long so it didn't completely ruin the show as long as you're not too anal. But if I fixed the file with VideoReDo the glitch was usually only about 1/2 a second and was less likely to effect the audio specifically which makes the glitch less burdensome.

    Through all these replies have you even bothered to install pyTivo Desktop? You might have a better understanding of what it can and cannot do if you gave it a try.
     
  3. ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    685
    60
    Jul 9, 2001
    Central Jersey
    I believe that, just from reading your posts, that I already possess a good understanding of what pyTiVo Desktop v1.6.1 can, and cannot, do. I was simply pointing out to you that your reply to my question was ambiguous and that someone less technical than you or I might expect pyTiVo Desktop v1.6.1 to work 'magic ' on their old, corrupt 'glitched ' TS / 'Fast' Format transferred files (i.e. I used @oldradio99's problem with his TS / 'Fast' Format transferred, 'glitched ' grandchildren's shows as an example).

    Give me some credit man :rolleyes: , I didn't just fall of the back of the turnip truck! :eek:

    Geez, after all of my support, ideas and encouragement, is that how little you think of me? Sad, so sad... :( (and disappointing...) Wow...
     
  4. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    You just seem pretty invested in offering ideas for a piece of software you haven't even tried. You should install it. I promise it wont make your computer blow up. :)
     
    Mikeguy, osu1991 and gonzotek like this.
  5. reneg

    reneg Member

    789
    22
    Jun 19, 2002
    I've spent some time in HxD looking at ts files and think I possibly have a way to build a clean ts file out of two or more downloads of corrupted ts files. If the corruption repeatedly shows up in the same area of each downloaded file, this will not work. I won't have time to prototype this into code for a while, but thought I'd share my thoughts, so Dan or others can tell me that it won't work.

    It's a bit sloppy, but for any of the following pseudo code to make sense, please reference the image I uploaded with this post.

    Code:
    Download Tivo File 1, convert to ts & scan with tivolibre, save output, if clean ts file (F1) then success & exit
    Repeat
        Download Tivo File 2, convert to ts & scan with tivolibre, save output, if clean ts file (F2) then success & break
    
        Determine in file F1 both P1 & Q1, and in F2 both P2 & Q2
        P1, P2 – First Point of corruption (“Invalid TS packet header for packet xxxx”), in HxD, first corrupted packet was xxxx minus 1
        Q1, Q2 – Start of clean data (“Resume decryption at: xxxx”),  packet number is xxxx/188
        If overlap between corruption areas (green) P1, P2, Q1, or Q2 Then break
    
        //Favor longer gap until first corruption in F1
        If P2 > P1 then swap IDs for F1<->F2, P1<->P2, Q1<->Q2
       
        In F2, start at F2.Q2 packet, find matching F1.P1 packet (minus 2, ie. last known good packet in segment A1) in F2, matching all 188 bytes of the F1.P1 packet (minus 2)
        if no match is found then break
        In F2, starting at F2.P1 packet, find matching Q1 packet in F2 matching all 188 bytes of the F1.Q1 packet.
        if no match is found then break
        If there are any corrupted packets between F2.P1 through F2.Q1 then break
       
        //Build a better ts file into F3
        Copy segment F1.A1 to new file, F3
        Append onto F3, from F2.P1 through but not including F2.Q1
        Append onto F3, from F1.Q1 to EOF
    
        Scan file F3 for re-sync with tivolibre, save output
        If no resync errors found then F3 is a clean ts file, success & break Else Set F3 id as F1 file, close F1
    Until Success or MaxAttempts reached
    
     

    Attached Files:

    ClearToLand likes this.
  6. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    In all my tests I find that the corruption is usually in the same spot. However I have found that with files that have two or more bits of corruption I will occasionally get a file that is missing one of them. So if you get lucky and and get two, or more, files without the corruption in each spot you could stitch them together and get a clean file.

    That being said I'm not sure tivolibre is going to do what you need. You can more easily do this in real code, even python. TS files have a sync byte every 188 bytes which is 0x47. So if you use buffers that are exactly 188 bytes aligned you can easily copy good chunk from one file over the bad part of another file.

    I think your best bet for logic would be to scan eash file as it downloads and keep a map of the bad chunks (location and length). If you get a duplicate of a file you already have delete it. Repeat as many times as needed until you have multiple files which can be merged into a clean file (or you simply get a clean file outright). When merging use the map to grab the chunks you need from each of the files and merge into one file.

    If you want to do this in the .tivo file, without decrypting, you'll need to skip past the .tivo header. To do that read the first 16 bytes into a buffer. Bytes 10-13 (0 based index) is a 64bit number corresponding to the size of the entire header. Do a seek of that value, minus the 16 you already read, and you'll be at the start of the TS file. (this byte should be 0x47)
     
    mlippert likes this.
  7. reneg

    reneg Member

    789
    22
    Jun 19, 2002
    Thanks for the feedback Dan. I went back and looked at the logs I uploaded earlier in this thread, and found that this would not have helped in any of the 8 downloads I performed on my Bolt. Multiple glitches per download and usually in the same spot. On my Roamio, it could have worked on the one file that couldn't get a clean download after 20 attempts. In fact, it would have been able to stitch together a clean file after only two downloads. If I do go forward with prototyping, maybe I'll just scale it back to take two glitched ts files as input and try and stitch them together into a clean ts file.
     
  8. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    I have one file on my Bolt that always has errors. It's a half hour and there is usually one at about the 10 minute mark and another around the 26 minute mark. I've downloaded this file hundreds of times now testing. Out of all those attempts I've only gotten a completely clean download twice. But I have had multiple occasions where I get a download with only one or the other and could have used those two files to create a clean download. The real question is how many times are you willing to try just to get a clean file?
     
  9. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    The new build of pyTivo Desktop with the TS error monitoring features has been posted
     
  10. lew

    lew Well-Known Member

    4,085
    39
    Mar 12, 2002
    I haven't bother participating. Not really an issue for me. I don't think am getting many errors. The 1/2 second glitch, after using VRD, isn't noticeable.

    Assume I don't bother with quick stream fix and directly edit the tivo file. Is the file similarly fixed when I use VRD to encode? Does it matter if I use the VRD software encoder vs Intel quick sync?

    FWIW Kevin said he tried throttling downloads and the errors persisted.
     
  11. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    If you recode in VRD it's doing essentially the same things as QuickStream Fix so the glitch should be oughly the same. In fact it might be even less noticable because it would fix any errors that the playback decoder might have due to missing reference frames. Some players are more sensative to this then other, which is why some will only show a minor glitch while others will skip over a few seconds of the video. With it recoded you'll get as many frames in the output as VRD is able to decode.
     
  12. mlippert

    mlippert Member

    99
    8
    Apr 2, 2010
    Massachusetts
    I've just been catching up on this whole thread. And thanks Dan, I haven't tried your new pyTivo desktop yet (I've been pretty happy w/ kmttg) but I may give it a shot soon. Thanks for all your work on this and your posts have been super informative.

    In regards to detecting the codec and using PS for mpeg2 and TS for h.264, I think that is a great idea, but I've got a suggestion that may be easier to implement (possibly).

    I don't think any channel switches how it's shows are encoded, so if you just let the user specify whether to use PS or TS for a particular channel, then it's a simple lookup to decide which to use with no need to download anything first. Add a default setting so that the user can only specify whichever are the exceptional channels. Even if I had to specify for every channel I download shows from, that list is 10-30 channels not that many.

    OR, if you want to do more work, build that channel list automatically, only checking the codec if the channel isn't in the list (or if it is PS and the download fails because the channel was recently changed to h.264. I don't think any channels are being changed from h.264 to mpeg2 so that isn't really worth checking) and then adding the codec you determined for that channel to the list.
     
  13. lew

    lew Well-Known Member

    4,085
    39
    Mar 12, 2002
    Dan. I don't know if the "magic" is done by vrd or by your encoder. Does it matter if we use the software encoder or can we use Intel quick sync and still correct glitches
     
  14. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    No. The encoder doesn't matter. The main difference between the way VRD fixes errors and the way tivolibre and pyTivo does is that when VRD detects an error it attempts to read the junk data and see if there is anything usable in there. tivolibre and pyTivo just skip all the junk data until the sync byte realigns.
     
  15. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    I might consider this. In the mean time I did add a feature in 1.6.1 where you can pick yourself on a per-recording basis. It will put a little TS and PS button next to each show and you can click whichever one you want for that show. There is also a menu added to the Download All button that allows you to pick TS or PS for that operation as well.
     
    mlippert and Mikeguy like this.
  16. tlp95129

    tlp95129 Member

    50
    0
    May 16, 2008
    California
    Dan,

    Love using your pyTivo Desktop. Great work.

    Recent glitch however. I'm on v1.62 now, and when I try to transfer a TS .tivo file from from PC to Tivo Roamio. my AVG-Free virus program flags tivolibre.exe and quarantines it. So the transfer fails. I ran a separate scan of the tivolibre.exe file in your bin directory and it passed OK. Has anyone else reported this? It's most likely bogus, but thought I'd report anyway.

    Link
     
  17. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    No one else reported. Just add the pyTivo install folder to its ignore list and you should be fine.
     
    Mikeguy likes this.
  18. elprice7345

    elprice7345 Member TCF Club

    178
    1
    Sep 28, 2009
    Houston, TX
    Not being that knowledgeable about video technical details, I’m having trouble following the TS error conversation.

    Comcast has switched most of my channels to h264, so I had to switch to TS transfers.

    I generally record shows, download them, and archive them until I pull them back to my TiVo to watch.

    Since I’ve switched to TS containers, sometimes the downloaded file has a shorter duration than the TiVo file. If I repeatedly download the file I can generally get the duration to match within 10 seconds or less.

    My current workflow:
    1. Download in TS container using kmttg
    2. Decrypt using VRD
    3. Run VRD QSF
    4. Check downloaded file duration vs. kmttg reported duration
    5. Re-download/process if the TS duration < kmttg duration
    6. Repeat until I get the file that most closely matches the kmttg duration
    7. Run VRD AdScan
    8. Cut commercials using VRD
    I’m OK with the durations not quite matching, assuming I’m missing a few seconds. Nothing “fails” in my workflow, just shorter files than expected.

    Do the failures only occur when downloading in TS containers, leaving in encrypted .tivo format and pulling back encrypted .tivo files?

    Does my workflow prevent this issue because of the decryption and/or QSF processing?

    What tool can I use to find TS sync errors?
     
  19. Dan203

    Dan203 Super Moderator Staff Member TCF Club

    40,550
    1,217
    Apr 17, 2000
    Nevada
    If you use my pyTivo Desktop to download them it will tell you if the file has TS sync errors. It also offers an option to retry download multiple times (user settable) until you get a clean copy or just keep the one with the least number of errors.
     
  20. lpwcomp

    lpwcomp Well-Known Member

    8,913
    161
    May 6, 2002
    John's...
    If you're using kmttg for this, then this is not actually 2 steps as VRD QSF is used for the decryption. If you're running QSF "manually", it is redundant.
     

Share This Page