New program for 1 step TTG downloads, decryption, encoding - kmttg

Discussion in 'TiVo Home Media Features & TiVoToGo' started by moyekj, Mar 15, 2008.

  1. Sep 29, 2019 #12361 of 12584
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    @moyekj (and now @davidblackledge ):

    AutoSkip from SkipMode [v] (to populate the kmttg AutoSkip Table with 'raw' SkipMode data) combined with Export AutoSkip cuts [e] to create a kmttg .EDL file are great features that I enjoy using in StreamBaby to provide hands off AutoSkip of programs offloaded to my NAS. @kearygriffin had added the ability of StreamBaby to use .EDL files BEFORE SkipMode was even thought of (possibly based solely on the available output from ComSkip). For 'Watch & Delete' shows, I don't see the need to invest CPU cycles to cut out commercials. But for movies and some shows that I keep to re-watch, like the six recently re-aired 3 hour Star Wars movies on TNT, we're talking about 45-50 minutes of commercials and several GB of HDD space.

    Note to non-StreamBaby users (i.e. PyTiVo, PyTiVo Desktop): Offloading shows to a NAS and then copying them back to a TiVo loses the SkipMode data because not all of the metadata needed gets returned to the TiVo for the TiVo to look it up again (ContentID IIRC). Using the available kmttg-generated .EDL file, StreamBaby automatically skips every commercial WITHOUT needing the kmttg AutoSkip Service running or using IFTTT. :cool:

    So, I recently revisited using the kmttg-generated .EDL file to bypass the ComSkip Ad Detect stage and was unsuccessful. ComSkip (under kmttg) recognizes the presence of the .EDL file but also wants a .TXT file. Even creating an icon for the ComSkipGUI.exe program and dragging the kmttg-generated .EDL file onto it fails. But, running ComSkip Ad Detect on another 30 minute show (the 3 hour Star Wars movies take 45-50 minutes to Ad Detect on my old laptop; then another 50-60 minutes to FFCUT.BAT) does create a ComSkip-generated .TXT (and an .EDL but with LFs instead of CRLFs) that contains the start and end markers in FRAMES instead of the SECONDS in the .EDL files. Once this .TXT file exists, with Ad Detect checked and Ad Cut not checked, I can exit and re-enter ComSkip Review at will as many times as necessary.

    Note to other ComSkip users: I find that by zooming out to where only one or two commercial blocks are displayed, cursor movement on a low-horsepower CPU is much improved.

    I researched this extensively and it appears that the ComSkip-generated .TXT file is the 'Master'. By changing values in the ComSkip.ini file and re-running ComSkip using the .TXT file as input, .EDL, .CSV, and (IIRC) .XML files can all be generated without rerunning Ad Detect but none of those files can be used to 'go backwards' to create the .TXT file.

    Initially I tried using NotePad but switched to NotePad++ to manually create my own ComSkip-simulated .TXT file from the kmttg-generated .EDL data because of the (now confirmed) double spaces after COMPLETE and AT and the 'invisible' TABS between the Start and End FRAMES numbers (use SPACES and ComSkip will ignore the line):
    Code:
    FILE PROCESSING COMPLETE  53915 FRAMES AT  2997
    -------------------
    18621    26433
    46560    52023
    53914    53915
    If I had access to MBASIC, or even Tiny Basic, I would take a shot at coding a little conversion program (just need to multiply the SECONDS by 29.97 and write out as FRAMES, separated by one TAB and ending with CRLF). But, with advancing age and poor health, I am not really set up for coding anymore. :( Analysis of the AutoSkip Table shows the 'raw' SkipMode data stored as HH:MM:SS.SSS and the routine to convert that into SECONDS already exists in the Export AutoSkip cuts [e] code. Add a 'Multiply by 29.97' and the Start and End positions are done; add a conversion to SECONDS Call for the total time and the Total FRAMES is done.

    I've proven that it works so if anyone would be so kind as to consider including it in a future version of kmttg I would be very appreciative. ;)

    Thanks very much from a big fan. :D

    P.S. By the lack of requests, I get the impression that VideoReDo *DOES* accept the kmttg-generated .EDL file as input and can thus skip the Ad Detect process and move on the 'Review' process. True?
     
  2. Sep 29, 2019 #12362 of 12584
    minimeh

    minimeh Vanity Phrase Here

    329
    71
    Jun 20, 2011
    San Francisco
    Not quite. Comskip generates a proper VRD project file (.vprj) with the ad skip data inside. By the way, it is comskip that generates the .edl file as well. Just depends on a command line parameter that kmttg sends to comskip.
     
    mlippert likes this.
  3. Sep 29, 2019 #12363 of 12584
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    Are you saying that if in File -> Configure -> VideoReDo, VideoReDo is enabled, when running Export AutoSkip cuts [e] a .VPRJ file is created INSTEAD of an .EDL file or in addition to?

    Two 'similar' .EDL can be generated by kmttg:
    1. One from Export AutoSkip cuts [e], where the data comes from the AutoSkip Table and the lines end in CRLF
      .
    2. One from ComSkip Ad Detect 'w' (i.e. write during Review), depending on the setting in Comskip.ini, where the data comes from the ComSkip .TXT file and the lines end in LF.
      If ComSkip is run INSIDE of kmttg, it won't run with an existing .EDL file; instead it will complain about a missing .TXT file.
      If ComSkip is run OUTSIDE of kmttg, it will run and it will overwrite the kmttg-generated .EDL file.
    If my update request is approved and added to kmttg, the kmttg-generated .EDL should be output for either ComSkip or VideoReDo users. In addition, a .TXT file should be generated for ComSkip users or a .VPRJ file should be generated for VideoReDo users.

    Make sense? [i.e. the VideoReDo users benefit from the SkipMode data; the ComSkip users don't.]
     
  4. Sep 29, 2019 #12364 of 12584
    minimeh

    minimeh Vanity Phrase Here

    329
    71
    Jun 20, 2011
    San Francisco
    Exactly.
     
    mlippert likes this.
  5. Sep 29, 2019 #12365 of 12584
    SamuriHL

    SamuriHL Member

    40
    10
    Jan 23, 2012
    Yes, that is correct. You can export the autoskip to vprj and "massage" the cuts in VRD. I've done this many times and it works well.
     
    mlippert likes this.
  6. Sep 30, 2019 #12366 of 12584
    billpiper

    billpiper Member

    66
    7
    Sep 17, 2015
    Richmond, VA
    Keep getting the following error for the last few days, Have rebooted and also powered down and restarted


    SkipService (Roamio): Attempting to re-connect
    rpc Read error - Connection reset
    rpc Read error - null
    SkipService (Roamio): Attempting to re-connect

    Auto skip functionality enabled
    kmttg ver 2.4n
    java ver 8 update 221
     
  7. Sep 30, 2019 #12367 of 12584
    davidblackledge

    davidblackledge Registered lÜser

    557
    35
    Sep 9, 2008
    NM
    Somebody else correct me if I'm wrong, but SkipService is the only part of KMTTG that's constantly polling your TiVos, so it's extra sensitive to your Roamio rebooting or your network having a spotty connection, or your network router/switch rebooting. Likely it's one of those things. I have a network switch in my house that is plugged into a loose outlet so any time it gets bumped that switch reboots.
     
  8. Sep 30, 2019 #12368 of 12584
    reneg

    reneg Well-Known Member

    1,100
    202
    Jun 19, 2002
    Feature Request - I've been using pyTivo Desktop to download sync error free TS shows from my Tivo and then using kmttg to process the rest of the file. I've been using ctrl-R on the NPL list in kmttg to get the metadata file. ctrl-R seems to always default to <filename>.tivo.txt. I then have to rename the file to <filename>.ts.txt. It would be great if ctrl-R in kmttg would allow me to get a <filename>.ts.txt file so I don't have to rename the file.
     
  9. Sep 30, 2019 #12369 of 12584
    billpiper

    billpiper Member

    66
    7
    Sep 17, 2015
    Richmond, VA

    Tivo plugged directly into router. No changes in connections in the last 3 years. have rechecked all connections and still getting the error
     
  10. Sep 30, 2019 #12370 of 12584
    elprice7345

    elprice7345 Member TCF Club

    346
    17
    Sep 28, 2009
    Denver, CO
    In a related enhancement request, could the multiple download attempt approach be added to kmttg?

    I use kmttg to download mpeg2 shows as .ps with no errors and some h264 channels download error free with kmttg as .ts downloads, but for most h264 channels (AMC, FXX, SYFY, etc.), I need to use pyTivo Desktop to repetitively download a show until I get an error free copy.

    It would be nice to see this approach added to kmttg so I can use one tool to process all my downloads!
     
    reneg likes this.
  11. Sep 30, 2019 #12371 of 12584
    ThAbtO

    ThAbtO TiVoholic by the bay TCF Club

    11,077
    787
    Apr 6, 2000
    SF Bay Area
    If the show is still on the Tivo, just Start Job with Metadata checked, then cancel the rest in jobs window.
     
  12. Sep 30, 2019 #12372 of 12584
    reneg

    reneg Well-Known Member

    1,100
    202
    Jun 19, 2002
    Doing that also starts a download. I'm downloading from the tivo in another session with pyTivo desktop. downloading two shows from the same tivo concurrently is no bueno. Ctrl-R on selected shows in the NPL in kmttg doesn't trigger a download. Ctrl-R also doesn't require me to cancel jobs. It's just not the desired file extension I would like.
     
  13. Sep 30, 2019 #12373 of 12584
    ThAbtO

    ThAbtO TiVoholic by the bay TCF Club

    11,077
    787
    Apr 6, 2000
    SF Bay Area
    Put the file in KMTTG's file tab and Metadata checked.
     
  14. Sep 30, 2019 #12374 of 12584
    reneg

    reneg Well-Known Member

    1,100
    202
    Jun 19, 2002
    Your suggestion didn't produce any results for me. Sounds like a chicken & egg problem for what I want anyway. I want the metadata file to use as a basis for naming the .ts file from pyTivo Desktop because my kmttg file naming template works with Plex.
     
  15. Oct 1, 2019 #12375 of 12584
    mlippert

    mlippert Active Member

    284
    58
    Apr 2, 2010
    Massachusetts
    @reneg I'm not sure how kmttg could do what you want except maybe by adding a configuration value for the what to use as the extension when downloading only the metadata.
    I thought kmttg renamed the .txt file as needed to stay consistent with each job step, but I think it must do some look-ahead.
    But if you download metadata followed only by downloading the show w/o decrypting or anything else, the encrypted file has the extension .TiVo, and the metadata file has the extension .TiVo.txt.

    If you decrypt the downloaded file then kmttg determines the format of the .tivo file and will produces either a .ts or a .mpg file and the metadata file will be named .ts.txt or .mpg.txt respectively.

    I believe the metadata file is again renamed if the video output file name is different after cutting out ads, and probably once again if the video is then encoded (I don't do this so I'm not sure).

    I suspect this is why the metadata only download gets the .TiVo.txt extension, there are no follow on jobs to determine how to rename it.

    That isn't to say that I wouldn't appreciate some change to address this as I also download (using my customized version of pytivo) and then let kmttg manage all the other processing (decrypt/qsfix w/ VRD, ad detect, and then later ad cut and create captions). While my customized version of pytivo has a naming template to be very close to kmttg, I haven't been able to get show season or episode numbers, and sometime the naming is a tiny bit off, so I need to download the metadata first.
     
  16. Oct 1, 2019 #12376 of 12584
    slowbiscuit

    slowbiscuit FUBAR

    4,409
    468
    Sep 19, 2006
    In the ATL
    As mentioned above this is likely a network error assuming you've rebooted the Tivo and restarted kmttg. Try moving the Tivo and PC to different ports on the router. I've had a GigE switch go bad where kmttg (and other devices on the switch) would get intermittent connection failed events. Swapped out the switch and been perfect ever since.
     
  17. Oct 1, 2019 #12377 of 12584
    reneg

    reneg Well-Known Member

    1,100
    202
    Jun 19, 2002
    I'm trying to stick to describing the problem as requested by @davidblackledge . I don't want to presume how such a feature could be implemented. It could be done with a configuration option, it could be done with a new hotkey. If the feature requested by @elprice7345 were implemented, my feature request would not be needed because I would no longer need to use pyTivo desktop to get clean ts downloads.
     
  18. Oct 1, 2019 #12378 of 12584
    davidblackledge

    davidblackledge Registered lÜser

    557
    35
    Sep 9, 2008
    NM
    I am sorry to all of you, but I'm not a big user of the different tools or even of kmttg, so I am likely to ask some dumb questions, such as this one. (It also means I'm not able to test all the combinations people encounter.)

    How do you know you have an error-free copy? is that a mysterious feature of pyTivo or there an obvious test for this? If the former, we'd have to look into pyTiVo for its technique and duplicate it.

    Nice information, thanks. In general, the metadata files are really important and should probably be managed more "globally" but that will take a lot of thought for how to rearrange things. In the meantime, it's nice to hear there is a possible "two birds with one stone" solution.
     
  19. Oct 1, 2019 #12379 of 12584
    ClearToLand

    ClearToLand Old !*#$% Tinkerer!

    1,064
    157
    Jul 9, 2001
    Central Jersey
    Looking at your TCF history, and going to the website pointed to by your TCF profile (and then the Wiki and then the Yahoo Group), I would have thought differently. You seemed *WAY* involved with ideas back a decade ago... :cool:

    Back in the spring of 2017, when @Dan203 was developing PyTiVo Desktop, once he mentioned the possibility of checking for TS Sync Errors, I became a dog with a bone. From my point of view, I felt that I was posting about "0x47 every 188" and "TS Sync Errors" obsessively and that anyone logged onto TCF would be aware of it. Sadly I came to realize that the folks here are highly "compartmentalized" (i.e. they come looking for the answer to THEIR problem and don't seem to read much more). :( There are some high post count TCF users (here and posting multiple times DAILY) who still don't distinguished between TS and PS downloads when instructing a new user how to back up (and then restore) a TiVo unit to a PC and one is still pushing TiVoDecode, which is the worst decrypting tool out of those available. Folks no longer talk about 'short' TS downloads (i.e. 57 minutes instead of 60 - @lpwcomp and @elprice7345 ); when I demonstrated how limiting bandwidth with a managed switch reduced TS Sync Errors, hardly a handful of folks took notice (@mlippert, @reneg, ???).

    If you or Kevin are interested in adding TS Sync Error Checking to kmttg, (if @Dan203 doesn't jump in), the comments in PyTiVo Desktop are pretty good:
    • Read in and buffer a good sized chunk, in 188 byte multiples, of the incoming TS download
    • After discarding the header (first 16 bytes IIRC)
    • Check the first byte of every 188 byte group for 0x47H
      If found, no error. If not found, increment TS Sync Error counter
    • If checking for "Least Number of Errors", continue to end and then re-start comparing Lowest Total Errors to Current Errors
    • If checking for "No Errors", abort and restart
    I'm not a Python or Java programmer, just a technical hobbyist. I know other, old languages so I tend to think in "pseudo-code". Thus I was able to hack PyTiVo Desktop to do a few things I wanted by searching the online manuals for the instructions to convert my "pseudo-code" into real code. I took a look at the kmttg download section. IIRC, it was just one command that said (basically) "Download this show"; no buffers; no loop. That would all have to be added. Feel free to contact me if you feel my input would be helpful.

    Although the three of us (@mlippert, @reneg, and myself) all have developed the routine of downloading the shows via PyTiVo Desktop in Transport Stream Format and then using either a kmttg-generated .EDL file or .TXT file to get our preferring naming convention (for Plex for example), I would tend to agree with @mlippert that kmttg currently acts as the standalone program that it was designed to be. It "thinks for itself" when controlling the 2nd to last extension of the metadata file, changing it dynamically during processing. I currently manually rename all my PyTiVo Desktop programs but I don't do this daily - just in spurts. Via GOOGLE last night, there are a few (free) Windows Renaming programs that I may eventually try. I'm also wondering if Windows Batch language could utilize "DIR /b *.TiVo.TXT" > Files.Txt to create a filename list that FINDSTR could parse for ".TiVo.Txt" and change it to ".TS.Txt" via RENAME. Just babbling...

    P.S. Any feedback on the possibility of adding the kmttg-generated ComSkip-compatible FRAMES counts .TXT file so that us ComSkip users can also take advantage of the SkipMode info? ;)
     
    davidblackledge and mlippert like this.
  20. Oct 1, 2019 #12380 of 12584
    Dan203

    Dan203 Super Moderator Staff Member TCF Club

    48,427
    4,861
    Apr 17, 2000
    Nevada
    It's pretty easy. You can request a specific amount of data from the TiVo on each call, so you just set that to a multiple of 188. Then you start at the beginning and ensure that every 188 bytes the value is 0x47. If it's not then you've got a corrupt packet. Just keep jumping 188 bytes until you detect 0x47 again. However many jumps you had to make to get there are the number of corrupt packets in the file.

    I tried various techniques of throttling the bandwidth in pyTivo to reduce errors, but none of them worked consistently. I also tried requesting really big chunks and really small chunks and that didn't seem to help either. Nothing I did in software ever seemed to really help reduce the number of errors. Although a few things I tried seemed to increase them.
     

Share This Page