TiVo Community Forum banner
1 - 20 of 3631 Posts

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #2 ·
I have a newer version available now (v0p1b) with some fixes and support for Linux (and likely other Unix flavors).
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #4 ·
I have a newer version available now (v0p1c) with some fixes and support for optionally running comskip.
comskip scans mpeg files for commercials and can spit out cut files which can be imported into popular editors such as VideoRedo, Womble, etc. to make it very easy to edit out commercials.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #6 ·
FYI VideoReDo has it's own commercial detection engine which is available via COM if you're interested in tapping into it.

Dan
Dan, that sounds very interesting, can you expand on that a little? I don't currently own VideoRedo, instead I have Womble Mpeg Video Wizard. With comskip it can create a .mls (bookmark) file which I can then import into Womble and spit out a modified .mpg with commercials removed. It's a very simple process, but only issue is I must manually bring up the Womble GUI to start the procedure - there is no command line way of doing it AFAIK.
If VideoRedo has command line capabilities for automating the whole process that is definitely something I would be interested in.
comskip has many methods of detecting commercials that can be tweaked. How accurate is the VRD commercial detection engine and are there different methods to select from? Womble has commercial detection capability as well but I find it doesn't work as well as comskip and there is not much tweaking that can be done.
Thanks.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #8 ·
I've never used comskip so I don't know how it compares to our ad detective. However iit can be completely automated like you want. In fact there is another user here on the forum who has written an app which uses it to automatically commercial scan and convert all .tivo files to .mpg...

http://www.tivocommunity.com/tivo-vb/showthread.php?t=359550

I don't believe it has the transcoding abilities of your program but perhaps you two should collaborate and come up with a solution that does it all.
Dan
OK thanks. I assume that if you have a cut file one can use it with VRD in batch mode (no GUI) to strip out commercials automatically? Assuming that is possible then one still would also have the option to use comskip instead of VRD ad detection.
This may push me over the edge to finally buy VRD. I just love the manual editing capabilities of Womble which has what has kept me from VRD.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #11 ·
Actually, for all practical purposes TiVo-VideoReDo-AutoProcessor (TVAP) already "does it all". It will automatically run a user-supplied batch file to perform the mpeg2-to-mpeg4 encoding using any encoding program with a command-line interface. HandBrake seems to work quite well and a sample batch file for XVID/AVI creation may be downloaded ***here***. The TVAP thread and the readme file give more details.

TVAP does not schedule TTG transfers however. You must use TiVo Desktop or another program to perform the transfers. I've toyed with the idea of adding the TTG transfer function to TVAP but operation with Desktop is pretty convenient so I haven't been too motivated.

I applaud your effort!
Looks like a great utility indeed for those using VideoRedo, great work! No sense me re-inventing the wheel there, so I think this program I will leave targeted at using freely available tools that run both on Windows and Linux. For my next update I will make some slight changes to make encoding phase more generic. Right now I based encoding narrowly on ffmpeg (it's really a great encoder), but there is no reason to narrow it down as such - let the user provide the executable name and arguments and kmttg will simply substitute the input and output file names. Same applies to the commercial removal tool.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #14 ·
I released a new version (v0p1e) which now allows specification of any encoder you wish instead of being hard coded to ffmpeg. The configuration file changed a little as a result to allow this.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #15 ·
I discovered today that "mencoder" can use a .edl file produced by "comskip" to automatically cut out commercials detected by "comskip", so my next version I'm integrating it into the flow so that the commercial cutting step is now also completely automated.
With this last change my goal of automating the entire process (downloading from Tivo all the way to re-encoding with option of having commercials removed) will be achieved.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #17 ·
Kevin,
Looks like fun. I've downloaded most of the files you reference, but I'm going to hold out until you finish the last change you mentioned. Any chance of you including the setup information for the full work flow? I read the installation notes, but I didn't see how comskip figures into it and I don't see a reference to mencoder. This is all a touch over my head - it's been years since I've done anything like this and I'm a bit rusty. Think there's any chance it can run on a Mac?

Thanks,

Anthony
I've released v0p1f which includes the automated commercial detection and removal and I've updated the config file and documentation appropriately. However be warned I did have a problem with 1 testcase where mencoder once it cut out commercials left the resulting video and audio out of sync. I'll have to test if Womble has the same problem with that video.

For windows I've included all the binaries you need and it's very easy to get up and running.

For Linux of course you have extra work to download and compile the 3rd party tools (namely tivodecode and ffmpeg and/or mencoder since curl is already part of most linux distributions). However I had no problems compiling them. The only exception is that comskip has no source code. People have reported success running it using "wine" on Linux.

For the Mac I would venture to guess it's not much harder to get working than on Linux.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #18 ·
I've released v0p1f which includes the automated commercial detection and removal and I've updated the config file and documentation appropriately. However be warned I did have a problem with 1 testcase where mencoder once it cut out commercials left the resulting video and audio out of sync. I'll have to test if Womble has the same problem with that video.
I did determine that for this testcase "mencoder" caused audo/video sync problems in the resulting mpeg while Womble did not. I also found that ffmpeg seems to run much faster than mencoder for re-encoding.

So based on the above findings I released v0p1g with following changes:
* Separated "comskip" from "comcut" steps so that user has option of using
the bookmark/cut files generated by "comskip" with another program for cutting out commercials.
* Now include both ffmpeg & mencoder in kmttg_win32_tools.zip
* In config_distrib.pl file I default ffmpeg as the default choice for encoder
* Added more notes on using kmttg in the README.txt file
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #21 ·
Kevin,
Almost worked! Everything up to recoding to the AVI. I got audio but no video. In the perl screen I got the following message:

Name "main::encoder_args" used only once: possible typo at c:\kmttg\kmttg.pl line 1260

Don't know if that's related. The mpg with the commercials cut was just fine, though. Any ideas?

Thanks,

Anthony
Anthony, no that warning is fine and expected as encoder_args is defined in a separate file (config.pl). Sounds like the encode step is working as there were no fatal errors. Could be you are missing a decoder to play it back on your PC? Are you using ffmpeg or mencoder for encoding? You can post here the entire command used for the encode - it is printed to the text area of kmttg when that step is performed. I can run using same options on a small clip to check if it works OK.

P.S. Next version I will make it possible to make text area text selectable so that copy/paste works (it already works in Linux as is but not Windows).
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #23 ·
Actually, right clicking on the text area brought up a dialog that let me copy the contents, so here's the relevant part:

>> ENCODING TO FILE c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4 ...
c:\kmttg\ffmpeg\rev11870\ffmpeg.exe -i c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20_cut.mpg -acodec libfaac -ab 128kb -vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 480x320 -title X c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4
c:\kmttg\Scrubs_My_Two_Dads_Thu_3_20.mp4 size=205.32 MB elapsed=00:11:48 (2.43 Mbps)

I changed the arguments to what the ffmpeg FAQ suggested for an ipod and modified to the correct resolution for a touch. So now I got an error 2010 - file contains invalid data. I'll have to study the syntax more to understand it. If it makes any sense to you I would appreciate your guidance.

Another question: Is there any way to bypass the download and find the files already on the computer and run everything from there? I've got Galleon auto downloading certain things that I would like to work with. Or better yet, any way to do what Galleon does - establish rules for what to automatically download? Hmmm... I guess the reason Galleon can do that is the server, so that won't work.

Thanks again,

Anthony
I think you are just missing -f mp4 option. The following in my config.pl works for me (I can play back resulting mp4 with VideoLan VLC player - windows media player doesn't have the codecs to play it back for me):
$encoder_args = "-i INPUT -acodec libfaac -ab 128k -vcodec mpeg4 -b 1200k -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 480x320 -title X -f mp4 OUTPUT";
$encodedFileExtension = ".mp4";

As far as bypassing downloads, you can sort of do what you want now with some workarounds. kmttg will skip steps depending on what files already exist in the output directory. Thus, for your example above, if Scrubs_My_Two_Dads_Thu_3_20.TiVo already exists, download is skipped. If you have decrypt enabled but Scrubs_My_Two_Dads_Thu_3_20.mpg already exists then it will bypass that step, etc.
One obvious problem is the file names kmttg uses are probably different than Galleon so you would have to re-name files accordingly.

I do have longer term plans to enable automatic downloads with kmttg based on title keywords, but that's probably a ways off...
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #25 ·
Sorry, kinda new with all this...

Does the Tivodecode that is used here decode the tivo file into an .mpg format that allows the HD Dvd authoring that is discussed in other thread on this forum?
Yes, the .TiVo file is decoded into an unencrypted mpeg2 format which you can then author to DVD or convert to other formats.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #27 ·
Kevin,
Did you change something in the step saving the mpg from the cut version? The last time (previous version) a 5 gb file trimmed down to around 4 gb. This time a 5 gb file trimmed down to 200+ mb. I have no problem with a smaller file size, I just wonder what happened.
No, nothing changed. 5GB->4GB sounds about right, 5GB->200MB is definitely not right and you should check the resulting _cut.mpg file as that seems way too small. Note that "mencoder" is being called to generate _cut.mpg file based on the .edl file generated by comskip, so it may be screwing up.
Is there any way to get the program to clean up after itself? Delete the working files like the .edl file? Heck, for my purpose I'd like to delete the .edl, the .tivo and the uncut .mpg leaving only the cut .mpg and the converted version.
Being conservative and still in development stage I thought it best not to delete anything along the way, however I will add an option to delete files as you go in the conversion process in a future release.
where is the naming convention set? As you pointed out in an earlier message, Galleon names things differently. I like Galleon's naming convention.
It is currently hardcoded in my Perl script. You could change it to suit your needs if you wish. I could add a config.pl option defining how to build the file name to make it more flexible but for now that's not high on my priority list.
Which program handles the download? Is there any way to have Galleon do that? Mind, I'm not asking you to do any of these things, unless you're of a mind to. I'm just curious about the possibilities. I did a bit of programming for active server pages in vbasic 10 or 12 years ago. You now have gotten me curious about perl.
I'm using "curl" for downloading shows from the Tivo (you can see the command being used printed in the text area). You can still use Galleon for downloads if you wish but then you have to deal with the naming convention problem if you then want to use kmttg for the rest of the flow. My whole intent of this project was to automate the process as much as possible and cut out manual tasks along the way, so injecting manual steps along the way kind of goes against that spirit.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #29 ·
Kevin, as a *seasoned* user of dlfl's TVAP program, I read with interest in what you're trying to accomplish. I wish you well.

One thing I like about TVAP is its ability to *pause* after running through VRD's Ad-Detective process allowing you to open a VRD (.vprj) project, make changes if needed then as soon as you Save the project, it continues about its business of creating a new file then deleting the original. Because I take a few minutes to verify the cuts, I now have the confidence to allow TVAP to create a new commercial-less file and delete the original automagically.

The reason I don't allow automatic cuts is that as good as VRD's Ad-Detective is, there are too many variables to how stations transition in and out of commercial breaks it's not accurate enough (100% of the time) for me to trust an automatic process of saving an edited file and deleting the original. To be fair, I've also used comskip for a while and found that it, too, wasn't accurate enough to automatically create a new file deleting the original. I know VRD will be enhancing their Ad-Detective process in the near future, but I think I will still *play it safe* and not let it perform the cuts automatically prefering to review them first.

Something to consider, at least.

With TVAP's post process batch file ability, I can convert the newly created commercial-less mpeg file, which I burn to DVD with VRD/TVS, to also create a mpeg-4 (.m4v for iPod) file via Handbrake automically. Two files for the price of one so to speak.

I like what you're trying to accomplish, but as Dan said earlier "perhaps you two [you and dlfl] should collaborate and come up with a solution that does it all." Good luck.
Thanks for your feedback. For precision commercial cutting I too don't rely on comskip or any other tool to do the job and prefer to do it manually (I do have cases when I don't really care about precision cuts when I am generating files for my Portable Video Player from a news recording). It is still useful to run comskip as a starting point and import that into a manual editor from which changes can be made.

Currently with the way kmttg is designed it will skip the download phase if the .TiVo file already exists. All the other steps are already optional. So to do what you propose the flow is:
1/ For 1st run enable "Decrypt .TiVo" and "Run comskip" options and run the program. This will create the .TiVo file and the unencrypted .mpg file and the comskip cut/bookmark file which can be taken into VRD.
2/ Take the cut file comskip produced and import to VRD and make changes as needed and then create the _cut.mpg file
3/ Run kmttg again this time with only the "Encode" option enabled to proceed with the encoding. Even if you do enable the earlier phases currently kmttg will skip them anyway if it sees the relevant files are already created.

i.e. All options currently in kmttg are optional except the download phase, so you can re-run kmttg and enable different options to run different portions, so a "pause" is not really necessary.

So, based on good feedback I am seeing here I think what is needed to make kmttg friendlier to that kind of flow is:
1/ Make the download step optional (as all other steps already are)
2/ Add an option (to File pulldown menu) to start processing from existing files instead of just from a Now Playing List. This way, for example, you can use your own .TiVo file names generated from other programs instead of adhering to kmttg names. Or if you already have have an unencrypted .mpg file with or without commercials, start from there. i.e. Allow the entry point to kmttg to be at any phase you want, which will also allow you to use whatever file names you want as a starting point.

Enhancement 1/ is very easy to implement. Enhancement 2/ will be trickier but I think possible.

I'm not against working with dlfl, however currently the emphasis of TVAP is using VRD and also we are using different 3rd party tools and programming languages, so there is not a lot in common as a starting point.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #34 ·
Thanks for the good feedback.

FYI. I am working on a new version which is getting close to release (it's tested on a linux box but I need to test using Windows).
In this new version there is an option to start processing from existing .TiVo or .mpg/.mpeg files anywhere on your computer instead of only Tivo Now Playing Lists. With this capability added this makes every step optional in the flow including download. I'm also going to convert the installation and useage instructions to HTML format to make it much easier to read.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #35 ·
I decided to bite the bullet and make the configuration GUI based as well to simplify things so will delay the next release a little so I can get that done.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #38 ·
Is there a way to get this running on Mac OS X?

Also, does it save metadata (original record date, guests, etc) with the file? Currently, I use the web based method (so I don't have to buy Toast), but it defaults to only the show name.. then I run tivodecode on it.
It already runs on Linux and I'm in the process of compiling detailed notes on how to get the 3rd party apps being used to built on Linux. I would imagine that getting it to work on Mac OS X should be possible if it's possible on Linux. However, I don't have a Mac to test with so somebody else perhaps can do the honors? I'd say wait for my notes on Linux installation and we can go from there.

As far as metadata right now I do parse all that data when reading the Now Playing Lists from the Tivo but I am not doing anything with them. It should be simple to write a file (perhaps in xml format) alongside the .TiVo file that contains all the information though as all the information is readily available. I will have to add this to my To Do List.
 

·
Registered
Joined
·
12,355 Posts
Discussion Starter · #39 ·
Okay, the anticipation is killing me! While I wait, is there a reference for the naming convention? I found the hard code in kttmg.pl, but I couldn't do much with it. For example, where are the underscores added in place of spaces? Is there a setting for the date format? Can I include the station it was recorded from (as Galleon does, when it dains to record something).

Thanks,
Anthony
As mentioned in my previous post all the extra information that goes along with the show including channel name, description, actors, etc. is all parsed already but I'm not doing anything with that data right now.

The buildTivoFileName function is the one building the file names in kmttg.pl and as you will see it strips out some special characters and substitutes spaces for underscores, etc. If you want to change file names that is the function to play around with. I do have plans in a future release to allow the name to be user customized but it's kind of low on my priority list right now.
You should be careful using the current release about leaving spaces in there, however, as it probably won't work right. In the new release I am working on I have quotes around all file names so it will be safe to have spaces in file names.
 
1 - 20 of 3631 Posts
Top