1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

New program for 1 step TTG downloads, decryption, encoding - kmttg

Discussion in 'TiVo Home Media Features & TiVoToGo' started by moyekj, Mar 15, 2008.

  1. Mar 22, 2008 #21 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    Anthony, no that warning is fine and expected as encoder_args is defined in a separate file (config.pl). Sounds like the encode step is working as there were no fatal errors. Could be you are missing a decoder to play it back on your PC? Are you using ffmpeg or mencoder for encoding? You can post here the entire command used for the encode - it is printed to the text area of kmttg when that step is performed. I can run using same options on a small clip to check if it works OK.

    P.S. Next version I will make it possible to make text area text selectable so that copy/paste works (it already works in Linux as is but not Windows).
     
  2. Mar 23, 2008 #22 of 10411
    Berone

    Berone New Member

    108
    0
    Feb 14, 2005
    Actually, right clicking on the text area brought up a dialog that let me copy the contents, so here's the relevant part:

    >> ENCODING TO FILE c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4 ...
    c:\kmttg\ffmpeg\rev11870\ffmpeg.exe -i c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20_cut.mpg -acodec libfaac -ab 128kb -vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 480x320 -title X c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4
    c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4 size=205.32 MB elapsed=00:11:48 (2.43 Mbps)

    I changed the arguments to what the ffmpeg FAQ suggested for an ipod and modified to the correct resolution for a touch. So now I got an error 2010 - file contains invalid data. I'll have to study the syntax more to understand it. If it makes any sense to you I would appreciate your guidance.

    Another question: Is there any way to bypass the download and find the files already on the computer and run everything from there? I've got Galleon auto downloading certain things that I would like to work with. Or better yet, any way to do what Galleon does - establish rules for what to automatically download? Hmmm... I guess the reason Galleon can do that is the server, so that won't work.

    Thanks again,

    Anthony
     
  3. Mar 23, 2008 #23 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    I think you are just missing -f mp4 option. The following in my config.pl works for me (I can play back resulting mp4 with VideoLan VLC player - windows media player doesn't have the codecs to play it back for me):
    $encoder_args = "-i INPUT -acodec libfaac -ab 128k -vcodec mpeg4 -b 1200k -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 480x320 -title X -f mp4 OUTPUT";
    $encodedFileExtension = ".mp4";

    As far as bypassing downloads, you can sort of do what you want now with some workarounds. kmttg will skip steps depending on what files already exist in the output directory. Thus, for your example above, if Scrubs_My_Two_Dads_Thu_3_20.TiVo already exists, download is skipped. If you have decrypt enabled but Scrubs_My_Two_Dads_Thu_3_20.mpg already exists then it will bypass that step, etc.
    One obvious problem is the file names kmttg uses are probably different than Galleon so you would have to re-name files accordingly.

    I do have longer term plans to enable automatic downloads with kmttg based on title keywords, but that's probably a ways off...
     
  4. Mar 23, 2008 #24 of 10411
    Prestond

    Prestond New Member

    44
    0
    Feb 4, 2008
    Sorry, kinda new with all this...

    Does the Tivodecode that is used here decode the tivo file into an .mpg format that allows the HD Dvd authoring that is discussed in other thread on this forum?
     
  5. Mar 24, 2008 #25 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    Yes, the .TiVo file is decoded into an unencrypted mpeg2 format which you can then author to DVD or convert to other formats.
     
  6. Mar 24, 2008 #26 of 10411
    Berone

    Berone New Member

    108
    0
    Feb 14, 2005
    Kevin,
    Still not getting the final conversion to work, but everything else went fine. Did you change something in the step saving the mpg from the cut version? The last time (previous version) a 5 gb file trimmed down to around 4 gb. This time a 5 gb file trimmed down to 200+ mb. I have no problem with a smaller file size, I just wonder what happened. That finished format will work well on my wife's laptop, which is one of the things I'm trying to achieve. But I don't know how it will translate going back to the Tivo, which is one of the other things I want - a commercial free backup that I can transfer back to my S3. The ipod format is gravy - I can always make that conversion in Nero as I have been doing.

    I'm really liking this program a lot - great work. It's more in line with what I want to do than anything else I've encountered. Couple of questions: Is there any way to get the program to clean up after itself? Delete the working files like the .edl file? Heck, for my purpose I'd like to delete the .edl, the .tivo and the uncut .mpg leaving only the cut .mpg and the converted version.
    Second question: where is the naming convention set? As you pointed out in an earlier message, Galleon names things differently. I like Galleon's naming convention. Last question: Which program handles the download? Is there any way to have Galleon do that? Mind, I'm not asking you to do any of these things, unless you're of a mind to. I'm just curious about the possibilities. I did a bit of programming for active server pages in vbasic 10 or 12 years ago. You now have gotten me curious about perl.

    Thanks,
    Anthony
     
  7. Mar 24, 2008 #27 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    No, nothing changed. 5GB->4GB sounds about right, 5GB->200MB is definitely not right and you should check the resulting _cut.mpg file as that seems way too small. Note that "mencoder" is being called to generate _cut.mpg file based on the .edl file generated by comskip, so it may be screwing up.
    Being conservative and still in development stage I thought it best not to delete anything along the way, however I will add an option to delete files as you go in the conversion process in a future release.
    It is currently hardcoded in my Perl script. You could change it to suit your needs if you wish. I could add a config.pl option defining how to build the file name to make it more flexible but for now that's not high on my priority list.
    I'm using "curl" for downloading shows from the Tivo (you can see the command being used printed in the text area). You can still use Galleon for downloads if you wish but then you have to deal with the naming convention problem if you then want to use kmttg for the rest of the flow. My whole intent of this project was to automate the process as much as possible and cut out manual tasks along the way, so injecting manual steps along the way kind of goes against that spirit.
     
  8. Mar 24, 2008 #28 of 10411
    msmart

    msmart New Member

    228
    0
    Jan 2, 2007
    Kevin, as a *seasoned* user of dlfl's TVAP program, I read with interest in what you're trying to accomplish. I wish you well.

    One thing I like about TVAP is its ability to *pause* after running through VRD's Ad-Detective process allowing you to open a VRD (.vprj) project, make changes if needed then as soon as you Save the project, it continues about its business of creating a new file then deleting the original. Because I take a few minutes to verify the cuts, I now have the confidence to allow TVAP to create a new commercial-less file and delete the original automagically.

    The reason I don't allow automatic cuts is that as good as VRD's Ad-Detective is, there are too many variables to how stations transition in and out of commercial breaks it's not accurate enough (100% of the time) for me to trust an automatic process of saving an edited file and deleting the original. To be fair, I've also used comskip for a while and found that it, too, wasn't accurate enough to automatically create a new file deleting the original. I know VRD will be enhancing their Ad-Detective process in the near future, but I think I will still *play it safe* and not let it perform the cuts automatically prefering to review them first.

    Something to consider, at least.

    With TVAP's post process batch file ability, I can convert the newly created commercial-less mpeg file, which I burn to DVD with VRD/TVS, to also create a mpeg-4 (.m4v for iPod) file via Handbrake automically. Two files for the price of one so to speak.

    I like what you're trying to accomplish, but as Dan said earlier "perhaps you two [you and dlfl] should collaborate and come up with a solution that does it all." Good luck.
     
  9. Mar 24, 2008 #29 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    Thanks for your feedback. For precision commercial cutting I too don't rely on comskip or any other tool to do the job and prefer to do it manually (I do have cases when I don't really care about precision cuts when I am generating files for my Portable Video Player from a news recording). It is still useful to run comskip as a starting point and import that into a manual editor from which changes can be made.

    Currently with the way kmttg is designed it will skip the download phase if the .TiVo file already exists. All the other steps are already optional. So to do what you propose the flow is:
    1/ For 1st run enable "Decrypt .TiVo" and "Run comskip" options and run the program. This will create the .TiVo file and the unencrypted .mpg file and the comskip cut/bookmark file which can be taken into VRD.
    2/ Take the cut file comskip produced and import to VRD and make changes as needed and then create the _cut.mpg file
    3/ Run kmttg again this time with only the "Encode" option enabled to proceed with the encoding. Even if you do enable the earlier phases currently kmttg will skip them anyway if it sees the relevant files are already created.

    i.e. All options currently in kmttg are optional except the download phase, so you can re-run kmttg and enable different options to run different portions, so a "pause" is not really necessary.

    So, based on good feedback I am seeing here I think what is needed to make kmttg friendlier to that kind of flow is:
    1/ Make the download step optional (as all other steps already are)
    2/ Add an option (to File pulldown menu) to start processing from existing files instead of just from a Now Playing List. This way, for example, you can use your own .TiVo file names generated from other programs instead of adhering to kmttg names. Or if you already have have an unencrypted .mpg file with or without commercials, start from there. i.e. Allow the entry point to kmttg to be at any phase you want, which will also allow you to use whatever file names you want as a starting point.

    Enhancement 1/ is very easy to implement. Enhancement 2/ will be trickier but I think possible.

    I'm not against working with dlfl, however currently the emphasis of TVAP is using VRD and also we are using different 3rd party tools and programming languages, so there is not a lot in common as a starting point.
     
  10. Mar 24, 2008 #30 of 10411
    dlfl

    dlfl Cranky old novice

    6,991
    16
    Jul 6, 2006
    Near...
    I have to agree. TVAP is written in C#. I am open to a cooperative effort but finding common ground seems pretty unlikely.
     
  11. Mar 26, 2008 #31 of 10411
    Berone

    Berone New Member

    108
    0
    Feb 14, 2005

    I found that ComSkip worked perfectly, assuming that what I actually wanted was information on hair loss. I reviewed the notes for tweaking ComSkip, but concluded that I could edit all the programs manually faster than get it set up to work most of the time. So that raises the question of manual editing: is there a freeware program available for video editing? I downloaded the trials of Womble and VideoRedo but they seem like overkill and a lot more than I want to spend to solve a not very important problem in my life. I looked at Window's movie maker, but... ugh! Nero's program doesn't seem interested in just cutting - it wants to make a whole new video.

    Yes, those options would do it for me. That and the automated downloads. When you get around to that one, if there is a way to select by other than title it would be great. Since Tivo's suggestions will frequently download reruns of shows that I have "first run only" season passes for, when Galleon is working (not much right now) it will download everything where I only want the first runs. Or the ones that aren't from the suggestion list, if there's a way to do that.

    Still haven't gotten it into mp4 yet, but your end of the operation is working. I just have to figure out the syntax. And then get it working on the Mac.

    Thanks!

    Anthony
     
  12. Mar 26, 2008 #32 of 10411
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Wow very cool! I prefer this idea over TVAP as it would allow me to use my linux server to handle everything.

    I just added some simple ToGo functionality to pyTivo last week. The possibility of adding your program to the backend is kinda cool.

    Keep up the good work.
     
  13. Mar 26, 2008 #33 of 10411
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    FYI comskip is available in source. The link is incredibly small on the page:

    svn://svn.kaashoek.com
     
  14. Mar 26, 2008 #34 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    Thanks for the good feedback.

    FYI. I am working on a new version which is getting close to release (it's tested on a linux box but I need to test using Windows).
    In this new version there is an option to start processing from existing .TiVo or .mpg/.mpeg files anywhere on your computer instead of only Tivo Now Playing Lists. With this capability added this makes every step optional in the flow including download. I'm also going to convert the installation and useage instructions to HTML format to make it much easier to read.
     
  15. Mar 27, 2008 #35 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    I decided to bite the bullet and make the configuration GUI based as well to simplify things so will delay the next release a little so I can get that done.
     
  16. Mar 27, 2008 #36 of 10411
    Berone

    Berone New Member

    108
    0
    Feb 14, 2005
    Okay, the anticipation is killing me! While I wait, is there a reference for the naming convention? I found the hard code in kttmg.pl, but I couldn't do much with it. For example, where are the underscores added in place of spaces? Is there a setting for the date format? Can I include the station it was recorded from (as Galleon does, when it dains to record something).

    Thanks,
    Anthony
     
  17. Mar 27, 2008 #37 of 10411
    mattack

    mattack Active Member

    20,734
    4
    Apr 9, 2001
    sunnyvale
    Is there a way to get this running on Mac OS X?

    Also, does it save metadata (original record date, guests, etc) with the file? Currently, I use the web based method (so I don't have to buy Toast), but it defaults to only the show name.. then I run tivodecode on it.
     
  18. Mar 28, 2008 #38 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    It already runs on Linux and I'm in the process of compiling detailed notes on how to get the 3rd party apps being used to built on Linux. I would imagine that getting it to work on Mac OS X should be possible if it's possible on Linux. However, I don't have a Mac to test with so somebody else perhaps can do the honors? I'd say wait for my notes on Linux installation and we can go from there.

    As far as metadata right now I do parse all that data when reading the Now Playing Lists from the Tivo but I am not doing anything with them. It should be simple to write a file (perhaps in xml format) alongside the .TiVo file that contains all the information though as all the information is readily available. I will have to add this to my To Do List.
     
  19. Mar 28, 2008 #39 of 10411
    moyekj

    moyekj Well-Known Member

    11,134
    27
    Jan 23, 2006
    Mission...
    As mentioned in my previous post all the extra information that goes along with the show including channel name, description, actors, etc. is all parsed already but I'm not doing anything with that data right now.

    The buildTivoFileName function is the one building the file names in kmttg.pl and as you will see it strips out some special characters and substitutes spaces for underscores, etc. If you want to change file names that is the function to play around with. I do have plans in a future release to allow the name to be user customized but it's kind of low on my priority list right now.
    You should be careful using the current release about leaving spaces in there, however, as it probably won't work right. In the new release I am working on I have quotes around all file names so it will be safe to have spaces in file names.
     
  20. Mar 28, 2008 #40 of 10411
    Berone

    Berone New Member

    108
    0
    Feb 14, 2005
    Kevin,
    Okay, I'm having fun with the file naming convention. Hopefully I'll figure it all out by the time you release the version making everything I'm doing obsolete! I tested it with spaces and the process worked (I'm just downloading and decrypting) but the decrytped version crashes Nero Showtime and itunes but works in WMP. So that's a problem! Work better pick up for me soon - I have too much time on my hands!

    One other thing for you to take a look at - better handling of errors when there are other transfers taking place. A couple of times I've gotten an error that says:

    <h2>Server Busy</h2>
    too many transfers in progress
    *ERROR* download failed (exit code: 0) - check above c:\kmttg\curl\curl.exe command

    Can't figure out what other transfers are in progress, but it would be nice if the program waited and retried rather than aborting.

    Anthony
     

Share This Page