TiVo Community Forum banner
  • TiVoCommunity.com Ambassador Program Now Open! >>> Click Here

New program for 1 step TTG downloads, decryption, encoding - kmttg

2M views 13K replies 921 participants last post by  mattack 
#1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 
See less See more
#2,845 ·
Question: What exactly does the Reset Tivo Web server do?
Most likely nothing useful. It executes following url:
Code:
http://<tivoip>/TiVoConnect?Command=ResetServer
That command was documented in HMO documentation (back when TiVo actually still had a web site with such information) but it most likely doesn't do anything useful, though the TiVo web server does accept the command.

I don't know what to tell you about stalled and/or slow downloads. With both my S3 OLED and Premiere units I've never had those kinds of issues (downloading from various OSs including XP, Vista 64, Win 7 64 and Linux) so I do not have conditions to debug anything along those lines. Apparently the web servers in THD units are more prone to having those issues perhaps because they are too underpowered. If you are getting very slow transfers and/or stalls I would check the more obvious things you have control over such as:
* Your network connections (wired is always better than wireless)
* Firewall and antivirus software, especially Kaspersky and the like can cause issues.
 
#2,846 ·
Thanks for the response.

As to the Reset Web Server. I sometimes use it but have no clue if it actually re-enabled transfer of it the server just revived its functionality over night on its own.

As to the stalled downloads: I made an experiment today in only downloading and decrypting 31 episodes of one show in SD quality. 22 of these 31 downloads have already completed w/o hickup while the Tivo did not record anything this afternoon. The test actually uses c:\cygwin\bin\curl.exe insteda of the 'java option' for download. Either SD or being 'idle' seems to help.

My next test will be to run only HD shows and see how this goes before I re-enable the qsfix, adscan and adcut again. I noticed that some VideoRedo3.exe tasks stay alive when I restart the service. Is it possible that kmttg does not purge some of its processes like:
cmd.exe /c ... VB script ...
when net stop kmttg is called ?

BTW: I am using Win 7 , connect Tivo and and Acer Revo via a 100MBit switch of a Buffalo router. Windows runs AVG and its own firewall.

Most likely nothing useful. It executes following url:
Code:
http://<tivoip>/TiVoConnect?Command=ResetServer
That command was documented in HMO documentation (back when TiVo actually still had a web site with such information) but it most likely doesn't do anything useful.

I don't know what to tell you about stalled downloads. With both my S3 OLED and Premiere units I've never had those kinds of issues so I do not have conditions to debug anything along those lines. Apparently the web servers in THD units are more prone to having those issues perhaps because they are too underpowered. If you are getting very slow transfers and/or stalls the only thing I would check are things you have control over such as:
* Your network connections (wired is always better than wireless)
* Firewall and antivirus software, especially Kaspersky and the like can cause issues.
 
#2,847 ·
My next test will be to run only HD shows and see how this goes before I re-enable the qsfix, adscan and adcut again. I noticed that some VideoRedo3.exe tasks stay alive when I restart the service. Is it possible that kmttg does not purge some of its processes like:
For something like VideoRedo note that is is really a 2 level deep command: a call to cscript which in turn initiates VideoRedo COM jobs. So by killing cscript alone this does not kill VideoRedo. For recent kmttg releases I use a scheme for the various VideoRedo scripts which monitors the presence of a file while it's running such that if the file is no longer present then VideoRedo exits. i.e. To kill a VideoRedo job it simply means deleting a file which causes VideoRedo and cscript jobs to exit. That's what happens under controlled circumstances. When shutting down service it may not be gracefully calling all appropriate kill procedures such as the above. In any case that should not be affecting downloads.

The fact that your SD downloads work very well and your trouble is with HD downloads does seem to point the finger at the TiVo having trouble doing it's conversions properly for TTG transfers - either glitches in the stream or too big of a load for processor to handle (since HD files obviously involve more processing than SD files on TiVo side).
 
#2,849 ·
Download failed to file: G:\TIVO3\Avatar (11_06_2010).TiVo
Exit code: 23

curl: (23) Failed writing body (2293 != 10416)

all files get same failure messages except last one in queue which succeeds.

If I set one of the failed files to be last in another queue it will succeed.

Had no problems for last three weeks.
 
#2,850 ·
Download failed to file: G:\TIVO3\Avatar (11_06_2010).TiVo
Exit code: 23

curl: (23) Failed writing body (2293 != 10416)

all files get same failure messages except last one in queue which succeeds.

If I set one of the failed files to be last in another queue it will succeed.

Had no problems for last three weeks.
Check volume G (disk space, etc). curl is not able to write all the data properly to that volume is the error being printed out.
 
#2,851 ·
Having issue with files being deleted after downloaded.

I do not have 'delete .tivo file after download' checked.

If I add just one file to the queue it downlads fine. If I add multiple files it deletes all but the last one after its downlads.

I did re-install java prior to this problem. Is this a java issue? has anyone had this issue?

I have all profiles removed and just download the tivo file to offload storage space. my only selection is the meta data.

Thanks.
It sounds like its overwriting the .tivo file as it downloads. Are you using the default filenaming scheme in your ocnfiguration?
 
#2,852 ·
Check volume G (disk space, etc). curl is not able to write all the data properly to that volume is the error being printed out.
I was using drive that was reformated fat32.:eek: I was running out of space on several externals and used this one forgetting that I changed it to fat32 for ps3 backup. New drive comes today.

donwloads now have no issues. Can not play older .tivo files using windows media player anymore. I found that video redo states that the media access key is wrong. .tivo files play fine on pc using tivo dekstop.

Its been a long week at work and not paying attention to detail when at home.

Is video redo worth purchasing to use for editing with kmttg?
 
#2,853 ·
I was using drive that was reformated fat32.:eek: I was running out of space on several externals and used this one forgetting that I changed it to fat32 for ps3 backup. New drive comes today.

donwloads now have no issues. Can not play older .tivo files using windows media player anymore. I found that video redo states that the media access key is wrong. .tivo files play fine on pc using tivo dekstop.

Its been a long week at work and not paying attention to detail when at home.

Is video redo worth purchasing to use for editing with kmttg?
FAT32 has a limitation of 4GB on files, I use NTFS on Win XP. Most, if not all, of my transfers from Tivo HD are greater than 4GB.

i use kmttg to transfer and decode to .mpg, then VideoReDo has no problems when I go to edit those commercials. Its well worth the cost. VideoReDo TVSuite also can make DVDs in single and double layer DVD-R's and DVD-RW's.
 
#2,854 ·
FAT32 has a limitation of 4GB on files, I use NTFS on Win XP. Most, if not all, of my transfers from Tivo HD are greater than 4GB.

i use kmttg to transfer and decode to .mpg, then VideoReDo has no problems when I go to edit those commercials. Its well worth the cost. VideoReDo TVSuite also can make DVDs in single and double layer DVD-R's and DVD-RW's.
Thanks. All past issues seem to be resolved. Look forward to try out editing shows this weekend.
 
#2,855 ·
I noticed something in regards to failing downloads:

While running kmttg v0p7q in GUI mode with Auto Transfers Loop in Gui enabled shows started to fail downloading again. The download was enabled via java not curl. After en/re enabling a few times with no effect I exited the GUI (via File->Exit)
The error messages were all of kind:
2010_11_17_19:55:26 ERROR: http IO exception for: http://192.168.0.20:80/download/Raising Hope.TiVo?Container=/NowPlaying&id=2209442
2010_11_17_19:55:26 ERROR: Server returned HTTP response code: 503 for URL: http://192.168.0.20:80/download/Raising Hope.TiVo?Container=/NowPlaying&id=2209442

which translates into: HTTP Error 503 - Service unavailable

Then I accessed the Tivo via Firefox and tried to download manually.
This gave the message:
Server Busy
too many transfers in progress

I restarted kmttg this time with no Auto Transfers Loop in Gui so it was idling

Since I have no other TTG client running I was sure that a process from kmttg must have been still running. Therefore in cygwin I issued: tasklist | grep java and found four java processes of which I killed the first three via
taskkill /f /pid <pid>

After this 'killing spree' kmttg began downloading again.

So my observation is that even when exiting kmttg from the GUI a stuck java download kept running. I have a suggestion although am not a Java programmer I do this in my shell scripting sometimes:
1) Is it possible to record the PID of each subprocess in a text file or the autlo.log.0 logfile ? That way once the user shuts down kmttg or its service it can be tracked if all processes are terminated
Example:
javadownload <args> <Showname> PID
or
curl.exe <arguments> <Showname> PID

2) Or can an exit handler be coded which takes care of leaving no trace behind?
3) Is there a way to restore functionality by decoding the HTTP error messages from the TIVO and then conditionally e.g. kill a download which is stuck / not making progress since it is blocking the queue?

Note: This program is so good that I consider learning java.
 
#2,856 ·
There already is a procedure that calls "kill" method for any job that is running if kmttg is closed, so under normal circumstances when you close kmttg there should not be anything left behind. It's easy to experiment with - for example start a "decrypt" job and then exit kmttg and then check if "tivodecode" is still running - it shouldn't be. Or switch to curl downloads and start a download and then exit kmttg and check if curl is still running. Depending on how kmttg is closed I suppose it may not be getting to the graceful exit which may be problem you are running into.
 
#2,857 ·
Started having a problem with kmttg last few days. When copying a program from my Tivo to my computer it stops at about the halfway mark.

I'm running Win 7 and kmttg v0p7k. I tried installing the latest ver of Java tonight but that didn't solve the problem. Thanks for any help.
Now I can't transfer anything at all via kmttg. It says "Warning: Transient problem: HTTP error. Will retry in 4 seconds." None of the retries are successful.

I tried via the web and it says "Server Busy. Too many transfers in progress."

I'm not transferring anything and my Bittorrent client is closed.
 
#2,858 ·
Now I can't transfer anything at all via kmttg. It says "Warning: Transient problem: HTTP error. Will retry in 4 seconds." None of the retries are successful.

I tried via the web and it says "Server Busy. Too many transfers in progress."

I'm not transferring anything and my Bittorrent client is closed.
Guess its time to reboot the Tivo.
 
#2,859 ·
So my observation is that even when exiting kmttg from the GUI a stuck java download kept running.
It was a rather obscure issue but I tracked down why kmttg JVM did not always kill all running process and exit properly when closing kmttg GUI as is supposed to be the case. I've checked in the change for next release. If you would like to test it out generate build from latest SVN or PM me for a beta copy with the fix.
 
#2,860 ·
Awesome,

moyekj, your turnaround time is unbelievable. As to using the new code: I have never built the code from svn myself but if you have a pointer with instructions I will give it a try. This way I can learn and hopefully at some point contribute somehow to this project.
Bug: So the actually bug in JVM only applies to exiting from running with the GUI but not when shutting down the service via net stop kmttg ? I am asking since usually I am running the service only and never observed this problem. But this week I was trying various options in the GUI and noticed the not-terminated processes.

It was a rather obscure issue but I tracked down why kmttg JVM did not always kill all running process and exit properly when closing kmttg GUI as is supposed to be the case. I've checked in the change for next release. If you would like to test it out generate build from latest SVN or PM me for a beta copy with the fix.
 
#2,861 ·
Yes, the bug was only exiting from GUI mode while tasks were still running. Only the 1st task in list would be killed and then JVM (javaw.exe in Windows) would hang.
I put together some instructions on how to build kmttg.jar from source:
http://code.google.com/p/kmttg/wiki/build_from_source

If you want to do Java development it's much easier with aid of a Java IDE kit such as Eclipse (what I use):
http://www.eclipse.org/downloads/moreinfo/java.php

Or just grab kmttg.jar build from latest source from here:
http://kmttg.googlecode.com/files/kmttg_v0p7r_beta1.zip
 
#2,862 ·
I've rebooted several times, it doesn't help with the kmttg problem. This reboot did solve the internet dl problem, but that is a tedious way to transfer these very large files. I would really like to solve my kmttg problem, I use this program a lot. The ability to transfer files to my computer is one of the main things I like about Tivo. I'd prefer not to have to go back to TivoDesktop if I can avoid it. Any suggestions appreciated.
 
#2,863 ·
I've rebooted several times, it doesn't help with the kmttg problem. This reboot did solve the internet dl problem, but that is a tedious way to transfer these very large files. I would really like to solve my kmttg problem, I use this program a lot. The ability to transfer files to my computer is one of the main things I like about Tivo. I'd prefer not to have to go back to TivoDesktop if I can avoid it. Any suggestions appreciated.
1st thing you should do is upgrade to latest kmttg version 0p7q. There was an update to kmttg to put a delay between multiple consecutive downloads to decrease the stress on TiVo web server. Also if you are running GUI make sure you also are not running kmttg service at same time.
 
#2,864 ·
1st thing you should do is upgrade to latest kmttg version 0p7q. There was an update to kmttg to put a delay between multiple consecutive downloads to decrease the stress on TiVo web server. Also if you are running GUI make sure you also are not running kmttg service at same time.
The upgrade seems to have solved the problem. I thought for sure I had checked to see if I had the latest version, maybe I went to the wrong place. Thanks much.

EDIT:

ARRGH! I spoke too soon, it's still happening. I checked my Services in Task Manager. There is something running called KtmRm, is that what you're talking about? Should I disable that?
 
#2,865 ·
ARRGH! I spoke too soon, it's still happening. I checked my Services in Task Manager. There is something running called KtmRm, is that what you're talking about? Should I disable that?
No, kmttg service is called just that - kmttg. Reboot your TiVo and PC one more time and then run kmttg v0p7q. Make sure you are not uploading videos to your TiVo or other network related tasks (like MRV, YouTube, etc) involving the TiVo while you transfer shows from it.
 
#2,866 ·
I followed your build instructions and noticed the behavior below from inside a windows xp command shell:

After having installed svn, jdj 1.6.0-22, apacha-ant-1.8.1
and doing
set ANT_HOME="C:\apache-ant-1.8.1"
set JAVA_HOME="C:\Program Files\Java\jdk1.6.0_22"

the ant command returns
"The syntax of the command is incorrect"

So I tried the same thing in cygwin

export JAVA_HOME="/cygdrive/c/Program Files/Java/jdk1.6.0_22"
export PATH=$PATH:/cygdrive/c/apache-ant-1.8.1/bin

cd /cygdrive/c/kmttg
ant

and that worked just fine

Any idea what the problem under a native windows shell might be?

Note: cygwin's svn version 1.6 did not work for me. The error message was:
svn: OPTIONS of 'http://kmttg.googlecode.com/svn/trunk': could not connect to server (http://kmttg.googlecode.com)

Yes, the bug was only exiting from GUI mode while tasks were still running. Only the 1st task in list would be killed and then JVM (javaw.exe in Windows) would hang.
I put together some instructions on how to build kmttg.jar from source:
http://code.google.com/p/kmttg/wiki/build_from_source

If you want to do Java development it's much easier with aid of a Java IDE kit such as Eclipse (what I use):
http://www.eclipse.org/downloads/moreinfo/java.php

Or just grab kmttg.jar build from latest source from here:
http://kmttg.googlecode.com/files/kmttg_v0p7r_beta1.zip
 
#2,867 ·
I followed your build instructions and noticed the behavior below from inside a windows xp command shell:

After having installed svn, jdj 1.6.0-22, apacha-ant-1.8.1
and doing
set ANT_HOME="C:\apache-ant-1.8.1"
set JAVA_HOME="C:\Program Files\Java\jdk1.6.0_22"

the ant command returns
"The syntax of the command is incorrect"

So I tried the same thing in cygwin

export JAVA_HOME="/cygdrive/c/Program Files/Java/jdk1.6.0_22"
export PATH=$PATH:/cygdrive/c/apache-ant-1.8.1/bin

cd /cygdrive/c/kmttg
ant

and that worked just fine

Any idea what the problem under a native windows shell might be?

Note: cygwin's svn version 1.6 did not work for me. The error message was:
svn: OPTIONS of 'http://kmttg.googlecode.com/svn/trunk': could not connect to server (http://kmttg.googlecode.com)
I use cygwin normally as well. However I did get it to work under native windows "cmd" shell as well. Take away the quotes in the set commands and then it works (at least it did for me).
i.e.
set ANT_HOME=C:\apache-ant-1.8.1
set JAVA_HOME=C:\Program Files\Java\jdk1.6.0_22
 
#2,868 ·
Thank you - removing the double quotes did the trick. Wondering why the shell then parses the white space in 'Program Files' correctly - well that is Windows weirdness.

I use cygwin normally as well. However I did get it to work under native windows "cmd" shell as well. Take away the quotes in the set commands and then it works (at least it did for me).
i.e.
set ANT_HOME=C:\apache-ant-1.8.1
set JAVA_HOME=C:\Program Files\Java\jdk1.6.0_22
 
#2,869 ·
Thank you - removing the double quotes did the trick. Wondering why the shell then parses the white space in 'Program Files' correctly - well that is Windows weirdness.
I think it is due strictly to the SET command behavior, with everything to the right of the equal sign being a part of that variable, including whitespace. I just did an experiment and found REMARKs are also included:

Code:
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

Tue 11/23
14:16:12=>set rem_comment=This is a new value REM Do I see this?

Tue 11/23
14:16:37=>echo %rem_comment%
This is a new value REM Do I see this?

Tue 11/23
14:16:50=>
 
Top