New program for 1 step TTG downloads, decryption, encoding - kmttg

Discussion in 'TiVo Home Media Features & TiVoToGo' started by moyekj, Mar 15, 2008.

  1. Oct 1, 2019 #12381 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    I also tried to create a technique that would stitch together several files to get one good one. But in my testing the vast majority of time the errors were in the exact same place, and if I did happen to get one that didn't have an error in one place it still had errors in all the same places as the other files so I never had enough good chunks to stitch together into a single good file. Usually if you get a file with less errors it's already got all the good spots anyway, so it's best just to keep that file. That's why I added the system that checks the number of errors in each file and keeps the one with the least number of errors after a certain number of tries.
     
  2. Oct 1, 2019 #12382 of 12622
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    Don't you then start the loop at byte 17, skipping over the header, and then check every 188 bytes?
    Did you ever try a Managed Switch or the NetLimiter program that @reneg uses?

    I use a Managed Switch and while a limit of 32 Mbps works fairly reliably for 30 and 60 minute shows, I needed to drop all the way down to 8Mbps for the six three-hour Star Wars movies recently re-aired on TNT. I also believe that file fragmentation on the TiVo unit influences the number of errors and I've tried "Permanently Deleting" ~100GB of shows from a TiVo in order to free up some contiguous space. I believe that TiVos that constantly operate in the 90-100% full state contain many fragmented files and an aging HDD may just take too long to seek to the next segment before the TiVo OS times out. Just a theory though...
     
  3. Oct 1, 2019 #12383 of 12622
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    @mlippert 's Python 3 version of PyTiVo has a very nice error report where you can plainly see the location(s) of the errors.

    I did a similar thing in your Python 2 version but I only wrote the errors out to the already-in-use log output and not a pretty-formatted report.

    If you do re-visit your version of PyTiVo Desktop, please give @mlippert 's error report a second look. :thumbsup:
     
  4. Oct 1, 2019 #12384 of 12622
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    Thanks for the reply @rocket999 .

    I'm more interested in understanding why the image of the kmttg webserver TiVo Remote moves up too high on my Fire Tablet in portrait mode than I am in installing a couple of 'VNC' programs to access kmttg a different way. If I could have modified an offset in a file to lower it, even if on both the portrait and landscape screens, I would have given that a shot. I also could give @wmcbrine 's TiVo Remote a shot but that would mean adding another program with a similar (sole) function compared to the multi-function kmttg.
     
  5. Oct 1, 2019 #12385 of 12622
    ggieseke

    ggieseke Well-Known Member

    5,531
    645
    May 30, 2008
    I still have yet to see even a single corrupt TS download out of more than 2,200 recordings. Maybe that's just because I only use TE3 and Roamios, but it's a significant number statistically.
     
  6. Oct 1, 2019 #12386 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    Yeah, forgot about the header. IIRC I read that separately as one big block, then I start reading the TS part in chunks with the error loop.

    No. To be perfectly honest I don't really use my own software that often. With so much available via streaming these days I find myself recording less and less. :/
     
  7. Oct 1, 2019 #12387 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    Are you downloading as TS or PS? This only applies to TS downloads. I think kmttg is set to use PS by default to avoid the errors. Although PS doesn't work with H.264 files and can have some weirdness with the captions, so it's not a perfect solution either.
     
  8. Oct 1, 2019 #12388 of 12622
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    I agree - I'm just trying to decide if you're a "Unicorn" or not? ;)

    I know we had a short discussion in another thread recently (weeks ago) regarding TS Sync Errors and *ACTUALLY* running @Dan203 's PyTiVo Desktop to monitor them (as opposed to someone just saying that kmttg, which doesn't check at all, isn't complaining and the user isn't 'seeing' anything). :rolleyes:

    I only use TE3 and I only record on my cablecard-equipped Roamio Basic and offload to a 500GB Roamio OTA and a 1TB Roamio OTA. If I can't keep up with the overflow, I then use PyTiVo Desktop to further offload to my NAS where I view the shows (with AutoSkip!) with StreamBaby. As I posted above, we used to get reports from @lpwcomp and @elprice7345 regarding "missing minutes" in kmttg Transport Stream Format downloads, but I can't name more than a half dozen or so folks here that care, or even know about, TS Sync Errors (and aren't already satisfied with using VideoReDo to 'cover them up' ;) ). Back when @Dan203 was developing PyTiVo Desktop, I was constantly getting shot down for my cheering "0x47 every 188" in most of my posts. :mad: I'm so glad he DID include it. :D

    Back to troubleshooting (other folks, not you) TS Sync Errors:
    1. Which Roamio models do you use?
    2. What size HDDs?
    3. What percent full are the HDDs? [thinking about disk fragmentation on the TiVo]
    4. Gigabit LAN?
    5. What are the specs of your PC? [Maybe my PCs aren't replying to the TiVo fast enough?]
    BTW, what TCF Forum do you think would be the best for a TS Sync Errors discussion?

    Personally, I would think it BELONGS in TiVoToGo but I feel that Coffee House and Help Center get more 'eyeballs' / views.
     
  9. Oct 1, 2019 #12389 of 12622
    ggieseke

    ggieseke Well-Known Member

    5,531
    645
    May 30, 2008
    TS only. About 60-70% of them are from my OTA basic Roamio and I actually prefer PS downloads in general, but it's easier to stick to one method. The rest of them are from my Roamio Pro on Comcast.
     
  10. Oct 1, 2019 #12390 of 12622
    mlippert

    mlippert Active Member

    289
    58
    Apr 2, 2010
    Massachusetts
    Isn't this the thread that was used to discuss TS Sync Errors?
    Reliable transfer of .TS files?

    Seems if there is more discussion to be had, that's a good place for it.

    And thanks for the call out @ClearToLand on my python 3 pytivo fork.
     
  11. Oct 1, 2019 #12391 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    Weird. I wonder why you have such good luck. In my testing I rarely got a file that didn't have at least one error. Maybe your network is slow and naturally falls into that sweet spot a few have achieved with throttling.
     
  12. Oct 1, 2019 #12392 of 12622
    ggieseke

    ggieseke Well-Known Member

    5,531
    645
    May 30, 2008
    I have only back-checked about 5% of those recordings for TS errors with Dan's pyTivo Desktop, but I have spent years editing them with VRD without seeing any corruption. I may be the unicorn in the house, but my gut says it's more about having everything on ethernet and not stressing my home LAN during downloads. I usually kick them off each morning and browse TCF while that's going on.

    The OTA recordings come from a basic Roamio with a 4TB MFSR drive. The cable recordings are from my Roamio Pro on Comcast with a factory 3TB drive. They have both been pushing 90% for years.

    I use a Technicolor modem from Comcast, but 99% of the traffic goes through a Linksys LGS116 16-port gigabit switch. The PCs involved are a Lenovo ThinkStation D20 on Win 7 that's about 12 years old, and a refurbished D30 from about 5-6 years ago that runs Windows 8.1. I crammed 256GB of RAM and a 24TB RAID5 array into the D30, so it isn't your average desktop.

    Dan's work is awesome and I wholly applaud his error checking on TS downloads, but somehow I seem to be immune to the need for it. If TS downloads aren't universally unreliable we need to ask why.
     
    ClearToLand likes this.
  13. Oct 1, 2019 #12393 of 12622
    davidblackledge

    davidblackledge Registered lÜser

    557
    35
    Sep 9, 2008
    NM
    Ah...I just looked at the tivolibre code...I am familiar with Frame Sizes and Sync Bytes...it's not exactly as your are describing (it actually slides forwards one byte at a time until it finds the sync byte again), but that's certainly a way to recognize an error occurred.

    I see that technically tivolibre can skip a packet that had a proper sync byte, but was flagged as an error packet by the TiVo. That tells me that I can't count on a number of bytes transferred vs. expected number as a definitive "it was bad."
    Really unfortunate is that TiVoLibre doesn't bother to provide a flag or count of any kind regarding the bad packets (or a flag to fail fast if a de-sync happens), considering it is doing this evaluation (in more detail) during its decode. I suppose if the user doesn't choose to decode kmttg still needs to detect the situation, though.

    So this all begs the next question: what to do when an error is detected. If the user has requested a guaranteed good TS transmission, will the TiVo play along if we kill the transfer on the first bad sync byte and try to start over? Or do we have to just complete the download, knowing it's a waste of time, just to keep the TiVo happy? Or is there value to keeping the corrupt file of which I'm not aware?
    My current thought is we would kill the connection and restart immediately if we even get one bad sync byte since all bets are off at that moment. This could be the very last packet, but that might be that last bit of the show you were counting on.

    Oh, I see now that Dan says if you get errors you're likely to keep getting at least some. So it does help to keep a "best copy" for when you give up. Dan, your definition of "good" should improve if you change it to a sliding window instead of a fixed width check.


    BTW, I'm curious if there is a BitRate argument for the download URL - I was noticing how there are filler packets tivolibre handles that are meant to maintain a constant bitrate. Maybe messing with such a URL argument might help with the error rate? Just a guess.
     
    ClearToLand likes this.
  14. Oct 1, 2019 #12394 of 12622
    davidblackledge

    davidblackledge Registered lÜser

    557
    35
    Sep 9, 2008
    NM
    I like this theory considering Dan's seeing errors in the same spot repeatedly for a given recording.
     
    ClearToLand likes this.
  15. Oct 1, 2019 #12395 of 12622
    mlippert

    mlippert Active Member

    289
    58
    Apr 2, 2010
    Massachusetts
    I actually had added my sync error logging specifically to see if the errors were consistently in the same place.

    For me at least, that wasn't true.

    Occasionally they were consistently in the same place, but I have it set to try downloading a maximum of 4 times, and frequently there was enough variation in those 4 attempts that if I could have taken pieces from several of them I could have put together a download w/o any sync errors. There certainly were times that that I couldn't do that w/ the 4 downloads, although frequently I could have done better than the best of the 4 when there were errors in all 4.

    For myself, 4 download gets an error free download a very large proportion of the time. Usually the amount of error blocks in the best one when I don't have an error free download is small enough that I don't worry about the loss when VRD cuts the error portion.

    Oh btw, my TiVos are almost always running very close to full 90-98%
     
  16. Oct 1, 2019 #12396 of 12622
    davidblackledge

    davidblackledge Registered lÜser

    557
    35
    Sep 9, 2008
    NM
    I mean... yes, I'm way involved...just not in the tools ;] I've spent almost all my time in the HME apps world and don't do a lot with processing my video files.

    I used to save movies off to a NAS but when that died, I switched to mostly streaming services for long term stuff, and TiVo capacity for the rest.
     
  17. Oct 1, 2019 #12397 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    Be careful with sliding check. That’s highly prone to false positives. When you're only working with a single byte sync like TS you'll find that 0x47 a lot in the actual data of the file too.

    In my experience the TiVo corrupts packets in multiples of 188 bytes. If they didn’t then my calculations would be way off because if it was shifted even 1 byte pyTivo would report all bytes after that as corrupt. Which it doesn’t.

    VideoReDo does a bit better that tivolibre at recovering corrupted frames from these files because it doesn’t just look at the TS sync byte, it looks for audio and video frame headers in the corrupted data and can still pick out good frames even if the TS wrapper is still corrupted. I didn’t want to go that deep in pyTivo so I just look for the corrupted packets.
     
  18. Oct 1, 2019 #12398 of 12622
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    49,099
    5,156
    Apr 17, 2000
    Nevada
    You could have tried it using a hex editor. Not sure how the encryption would work though. I think it uses a rolling key type system so it might have failed. I never got so lucky. I had a test file I was working with and I downloaded it dozens of times. A couple times I got a clean copy, but every other time the errors were always in the same exact spot. (down to the byte)
     
  19. Oct 1, 2019 #12399 of 12622
    mlippert

    mlippert Active Member

    289
    58
    Apr 2, 2010
    Massachusetts
    I actually thought about that, and I added configuration to be able to save all the downloads to try it.
    However the 1st thing I did was diff the files, hoping that the uncorrupted sections would be identical. That wasn't the case :-(. I think there's a new salt for each download or something like that. So I'd have had to decrypt all packets of each download 1st and then reconstruct the file. It didn't seem worth the effort to me.
     
  20. Oct 1, 2019 #12400 of 12622
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    I view this as a slight tangent to both get @davidblackledge up to speed on the topic *AND* to possibly make more (of the seemingly 'compartmentalized') TCF folks aware of TS Sync Errors.
    Has it gotten any more traction lately? (I haven't accomplished ANYTHING in the past two years! :( I hope to resume having fun AFTER my heart surgery.)
     

Share This Page