TiVo Community Forum banner
  • TiVoCommunity.com Ambassador Program Now Open! >>> Click Here

New program for 1 step TTG downloads, decryption, encoding - kmttg

2M views 13K replies 921 participants last post by  mattack 
#1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 
See less See more
#2,420 ·
Gremlin,

Thanks.

I updated Java today, using Software Update, and 'java -version' in Terminal shows that I now have Java version 1.5.0_24 on my Mac OS X 10.5.8 iMac. I was running 1.5.0_22 before. kmttg's main process/window is still experiencing bursts of seeming inactivity lasting several seconds at a time (though I think the sub-processes for its various "jobs" -- curl for doing downloads, tivodecode, etc. -- are running OK).

I have to assume that Java 1.6.n is not available unless I upgrade to Snow Leopard aka Mac OS X 10.6? Is that your understanding, too?
I just did Software Update again and my 10.5.8 system also still has Java 1.5.0_24. So I don't think you can "officially" get to Java 1.6 w/o upgrading to Snow Leopard. There may be some unofficial ways of updating just the Java software in your computer, but that's way beyond my OS X hacking abilities, so I can't advise you about that.
 
#2,421 ·
I just did Software Update again and my 10.5.8 system also still has Java 1.5.0_24. So I don't think you can "officially" get to Java 1.6 w/o upgrading to Snow Leopard. There may be some unofficial ways of updating just the Java software in your computer, but that's way beyond my OS X hacking abilities, so I can't advise you about that.
Gremlin,

Thanks. Does kmttg using Java 1.5.0_24 on your 10.5.8 system have any issues like I'm experiencing with short bursts of activity interspersed with unexplained "dropouts" in responsiveness, etc.?

Today I restarted kmttg from scratch and told it to save metadata, download, decrypt, and extract closed captions for a 9 GB recording on a TiVo Premiere that uses a wireless-N network connection. This was at 10:37 AM. It's now 8:34 PM and the decryption job is crawling along, about 2/3 done. The download took over 5 hrs., from 10:37 AM to 3:59 PM!

I did not have any other kmttg jobs queued during this entire time.

I'm pulling my hair out ... all this used to work fine.

I'm tempted to do a clean install of OS X 10.5 next and see whether my problem gets cleared up by that!
 
#2,423 ·
Sounds to me like you have disk I/O problems. If you take that 9GB recording and copy it to a different file name on same disk does it also take much longer than you would expect?
moyekj,

Excellent guess!

I am currently trying making a duplicate of my 9GB file to the same external drive. It's taking roughly 50-60 minutes to wrap up. I calculate its overall transfer rate at about 3MB per second -- though of course it is reading and writing data, so maybe I should double that.

Whether or not I double it, it's way below the nominal 480MB/sec of USB 2.0, which is the interface the external drive uses.

I need to look into what's wrong here. I've got five external hard drives (and a printer and SoundSticks) on two 4-way USB 2.0 hubs hooked to my Mac. One of the hubs is new, as is one of the drives. Three of the drives, including the newest one, are e-SATA drives sitting in BlacX docking stations. This general arrangement has been working fine for me, but maybe the new hub, or drive, or docking station has caused something to break. Further experimentation is in order, after which I'll report back ...

Edit: OK, I found out what the problem was. Oops ... I was nearly out of space on the external drive! Any time I would try to use that drive, kmttg would have short bursts of activity interspersed with longer periods of non-responsiveness. I didn't notice how low on drive space I was, nor did I notice that kmttg has a config setting for Min Requested Space (GB) that, had I been using it, would have alerted me to the situation.

Thanks for your excellent input!
 
#2,424 ·
Edit: OK, I found out what the problem was. Oops ... I was nearly out of space on the external drive! Any time I would try to use that drive, kmttg would have short bursts of activity interspersed with longer periods of non-responsiveness. I didn't notice how low on drive space I was, nor did I notice that kmttg has a config setting for Min Requested Space (GB) that, had I been using it, would have alerted me to the situation.

Thanks for your excellent input!
I'm so glad that you've got it sorted! It's summer here, so I have not been able to spend time on this. I was going to report back that I have a "Leopard Server" running Mac OS X 10.5.8, with Java 1.5.0_22, and then I've also got a "Snow Leopard Server" running Mac OS X 10.6.0, with Java 1.6.0_15.

I was also going to have you quit everything that has to do with kmttg, and try the curl command by itself, to see if that raises your issue to the surface again. (It would have ruled out kmttg and the distributed pieces, since curl is embedded in the OS)

The other thing I would have mentioned is that I'm still back on the "j" version of kmttg... (For the same reason above, it's summer and I haven't got much time to fool with this.)
 
#2,425 ·
I'm so glad that you've got it sorted! It's summer here, so I have not been able to spend time on this. I was going to report back that I have a "Leopard Server" running Mac OS X 10.5.8, with Java 1.5.0_22, and then I've also got a "Snow Leopard Server" running Mac OS X 10.6.0, with Java 1.6.0_15.

I was also going to have you quit everything that has to do with kmttg, and try the curl command by itself, to see if that raises your issue to the surface again. (It would have ruled out kmttg and the distributed pieces, since curl is embedded in the OS)

The other thing I would have mentioned is that I'm still back on the "j" version of kmttg... (For the same reason above, it's summer and I haven't got much time to fool with this.)
Thanks for your concern and input, AudioNutz. Actually, I was not aware that curl can be executed in standalone fashion! That's what comes of not looking hard enough at the messages output by kmttg. Right at this moment there's the following message showing a curl call by kmttg:

Code:
/usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie2423857590188571200.tmp --url http://10.0.1.7:80/download/The%20Story%20of%20India.TiVo?Container=%2FNowPlaying&id=1601172 --output "/Volumes/Hitachi 2TB Drive/Downloads/The Story of India - Beginnings (06_11_2010).TiVo"
I'm not sure what all those options do, but clearly --url specifies the URL of the particular TiVo file I'm downloading, complete with the IP address my TiVo is currently using, 10.0.1.7. --output gives where the output .TiVo file is to be put. --user tivo:MAK gets filled in, I assume, with my media access key.

Anyway, I've learned my lesson: pay attention to how low on space my destination hard drive is getting!

Cheers,
 
#2,427 ·
I was also going to have you quit everything that has to do with kmttg, and try the curl command by itself, to see if that raises your issue to the surface again. (It would have ruled out kmttg and the distributed pieces, since curl is embedded in the OS)
The problem has been solved, so this is a suggestion for others who may stumble upon this thread when searching ...

In addition to curl, one other technique is to use a browser to go directly to https:<tivo_ip_address> and download that way. IIRC the username should be tivo and the password should be your MAK.

The idea is that when more complicated programs like kmttg have problems, try simpler techniques, it makes troubleshooting easier.
 
#2,428 ·
I've noticed that kmttg removes the punctuation marks from the title of recordings it downloads -- apostrophes, question marks, and so on. I wonder whether there shouldn't be a config option to prevent this? It's a pain putting the punctuation marks back in, and, after all, only ':' isn't allowed in file names. Thanks for listening ...
 
#2,430 ·
Unless you are running Windows...
I'm on a Mac. The situation with Windows is why I suggest that leaving punctuation characters as is ought to be user-configurable. Or, alternatively, kmttg could check to see what operating system it's running on, and act accordingly ...
 
#2,432 ·
This may have been asked before, in case it has, I apologize.
What is the trick in FFMPEG to force voice synchronization with the movie.
I inevitably end up with the voice about a second out of synch by the end of encoding.
I use the following command line that works great for the low volume problem I had on my Archos player, but the voice does get out of synch (which I thought -async 1 prevented). It is still watchable, but irritating.

FFMPEG -y -i INPUT -threads CPU_CORES -vcodec mpeg4 -maxrate 2000k -qmin 3 -qmax 5 -bufsize 5096k -g 300 -acodec libmp3lame -ac 2 -ab 128k -ar 48000 -vol 1280 -s 720x480 -r 29.97 -b 1500k -async 1 -f mp4 OUTPUT

Thanks
Update:
I saw this on a Linux site, so maybe I do need another value:
-async 1 is a special case where only the start of the audio stream is corrected without any later correction.

Does anyone know a fool-proof number for async?
 
#2,433 ·
Even on the Mac, kmttg makes calls to several "Command Line" tools. These are not always friendly when it comes to punctuation. If you read back in this thread far enough, I'm sure you'll find the part where this issue was discussed.
OK, I'll look for those past posts. I did do a search in the thread on "punctuation" but got no results.

But I also have to note that there is really no problem at all on the Mac command line with file names containing punctuation. Just enclose the whole, fully qualified file name in double-quote characters. Then, the only restrictions become those that the Mac OS itself imposes, e.g., '/' characters are not allowed in file names. Neither are ':' characters in the Finder — as the two mean the same thing, with '/' in the command line and ':' in Finder both representing the separator between directory levels or between the directory and the filename proper.

I have to believe that kmttg already puts double quotes around filenames, since they're the easiest way to allow spaces to appear in a filename. So I imagine kmttg is simply filtering out such characters as apostrophe and question mark to cater to OS's that don't permit them. So why not make that filtering process configurable? On a Mac, the user could optionally choose to bypass it. Users on other OS's could leave it alone. No harm, no foul ...
 
#2,434 ·
...But I also have to note that there is really no problem at all on the Mac command line with file names containing punctuation. Just enclose the whole, fully qualified file name in double-quote characters. Then, the only restrictions become those that the Mac OS itself imposes, e.g., '/' characters are not allowed in file names. Neither are ':' characters in the Finder - as the two mean the same thing, with '/' in the command line and ':' in Finder both representing the separator between directory levels or between the directory and the filename proper...
So on a Mac, you can have a path that looks like this?:
C\Path\To\Some\Folder\File.extension

and/or this?:
C:path:To:Some:Folder:File.extension

I'm sure my Mac ignorance is blatantly showing... :eek:
 
#2,435 ·
So on a Mac, you can have a path that looks like this?:
C\Path\To\Some\Folder\File.extension

and/or this?:
C:path:To:Some:Folder:File.extension

I'm sure my Mac ignorance is blatantly showing... :eek:
/Path/To/Some/Folder/File.extension

is valid on the Mac. The separators are forward slashes, not backward slashes. The first '/', starting from the left, stands for the root directory, as understood by Unix.

There are no drive letters, either. You specify which hard-drive volume to use with something like:

"/Volumes/New 2TB Drive/TiVo Transfers/Just Transferred/Mr. Hollands Opus (06_23_2010).TiVo"

Instead of a drive letter, the whole thing starts with /Volumes/New 2TB Drive/... .

(Notice that the name of the movie has been changed by kmttg to "Mr. Hollands Opus," with no apostrophe. That's what I'm basically objecting to.)

The syntax containing ':' characters is not actually used on the command line, but it is not allowed by the Finder, which is sort of the Mac's equivalent to the whole gamut of Windows windows that show the contents of directories.

Get it? The Finder is part of the Mac OS X operating software, but it is not itself Unix. Mac OS X (and Finder) sit atop Unix, which is what you are dealing with in the command line itself.

So you never see any syntax with ':' characters, anywhere. But Finder internally turns Unix '/' characters into ':', so it won't let you embed ':' in visible filenames.

It sounds complex, but as long as kmttg were to put the filenames all in double-quotes and avoid internal double quotes, '/' characters, and ':' characters, fully qualified Mac filenames would come out fine.
 
#2,436 ·
I'll let Kevin chime in here. I think the apostrophe's were getting replaced with some other character automagically during the download process, because of some issue with curl. The remaining commands were attempting to execute with the correct name with the apostrophe, but since the name was truncated by curl... It couldn't find the file to perform the work on.
 
#2,437 ·
I'll let Kevin chime in here. I think the apostrophe's were getting replaced with some other character automagically during the download process, because of some issue with curl. The remaining commands were attempting to execute with the correct name with the apostrophe, but since the name was truncated by curl... It couldn't find the file to perform the work on.
OK, thanks, I've found the post that talks about it, #487 in this thread. (Is Kevin moyekj, BTW?)

So, is curl really the critter responsible for dropping the apostrophes? (And, I assume, other characters such as question marks ... ) I ask because I recently downloaded a program called

Closer to Truth - Why Is Consciousness Baffling?

that has a question mark. I saw in kmttg's messages:

Code:
Closer to Truth - Why Is Consciousness Baffling?
Recorded Mon 06/21/2010 08:30 AM on 268=WMPTDT2, Duration = 30 mins
Understanding inner awareness. 
OVERWRITING EXISTING FILE: /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt

 ...

>> CREATING /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt ...
/usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie2199505577978024301.tmp --url https://10.0.1.7:443/TiVoVideoDetails?id=1651715 --output /tmp/meta8375990166995294744.tmp 
metadata job completed: 00:00:01
---DONE--- job=metadata output=/Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt

 ...

>> DOWNLOADING /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).TiVo ...
/usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie8252577655988769831.tmp --url http://10.0.1.7:80/download/Closer%20to%20Truth.TiVo?Container=%2FNowPlaying&id=1651715 --output "/Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).TiVo"
That OVERWRITING EXISTING FILE message suggests to me that the '?' was dropped before curl was called!

If the same logic applies to apostrophes, then are we sure the apostrophes disappear only at the time curl is called?

Of course, it may be the case that the punctuation characters have to be made by kmttg to vanish at (or prior to) the curl call simply to get curl to work ... but, if so, is there any reason they can't (optionally, at user discretion) be stuck back into the filename by kmttg after curl has done its duty and the downloaded TiVo file is sitting there on the hard drive? I.e., can't kmttg optionally rename the TiVo file, once it's been downloaded, and use the revised name that contains all the original punctuation characters in subsequent jobs such as decrypt.

I note also that the apostrophe in "Mr. Holland's Opus" and the question mark in "Why Is Consciousness Baffling?" are preserved in the metadata files created by kmttg.

Of course, a lot of what I'm saying is guesswork on my part, and I don't really have any idea how easy or hard it would be to give Mac users of kmttg a "preserve punctuation" option implemented via file-renaming logic. In view of the fact that kmttg is multi-platform software, my idea might adversely impact users of other platforms. It might just be too hard to implement my idea without going to a lot of trouble to re-code affected areas within kmttg, so as to enable this "preserve punctuation" option that would cater to only a small subset of users anyway.

Anyway, that's my two cents ...
 
#2,438 ·
epstewart, special characters in file names for Unix/Mac systems is not as trivial as you may think to support, especially in the context of making system calls via Java. kmttg basically uses Runtime.getRuntime().exec(String []) to execute system calls and characters such as single or double quotes, slashes, backslashes and a few others in that context do cause issues. Probably with quite a bit more work and some OS-specific considerations the allowed character list per OS could be expanded, but it's really a can of worms I don't feel like opening. The list of characters that are filtered out now were mostly born out of past issue reports.

If you feel strongly about it then source code is available to make whatever changes you may want to try. The place to start looking would be removeSpecialChars method in main/tivoFileName.java. This is what filters out characters in file names before any task in kmttg is called (so yes the characters are stripped out before calling curl, etc).
 
#2,439 ·
epstewart, special characters in file names for Unix/Mac systems is not as trivial as you may think to support, especially in the context of making system calls via Java. kmttg basically uses Runtime.getRuntime().exec(String []) to execute system calls and characters such as single or double quotes, slashes, backslashes and a few others in that context do cause issues. Probably with quite a bit more work and some OS-specific considerations the allowed character list per OS could be expanded, but it's really a can of worms I don't feel like opening. The list of characters that are filtered out now were mostly born out of past issue reports.

If you feel strongly about it then source code is available to make whatever changes you may want to try. The place to start looking would be removeSpecialChars method in main/tivoFileName.java. This is what filters out characters in file names before any task in kmttg is called (so yes the characters are stripped out before calling curl, etc).
Thanks for a very helpful reply. I gather, then, that the difficulty in making system calls via Java using strings with pesky special characters in them is the root cause for the need to suppress those characters in output filename literals.

I have no knowledge of writing Java code, so I am totally naive here. Still, I am wondering if there is a way to "escape" special characters such as ? and ' in the literals in question. As you may know, in Unix the backslash (\) is generally used to designate that the following character is to be taken literally, as part of the string itself. So \" allows a double quote to be included in a string that is delimited by " characters, for example. Likewise, backslash can be included via \\. Using such escape sequences, any character can be included in any literal string.

I would think there must be a way to do the same kind of thing for strings used in Java system calls, no?
 
#2,440 ·
As you may know, in Unix the backslash (\) is generally used to designate that the following character is to be taken literally, as part of the string itself. So \" allows a double quote to be included in a string that is delimited by " characters, for example. Likewise, backslash can be included via \\. Using such escape sequences, any character can be included in any literal string.
It's not unix, it's the shell that does that.

When I run bash or tcsh (or whatever your favorite shell is) on my windows system (under cygwin, for example) I can do the same thing when I type those characters, but it's the shell that processes them. They don't get passed into library calls or system calls.

-David
 
#2,441 ·
Yes, it's a very important point by Icarus - system call from Java does not invoke a shell. You can choose to invoke the shell as part of your system call but that just adds another layer of complication. So the usual shell escape parameters do not necessarily apply.
 
#2,442 ·
Awesome instructions for setting up pyTivo here: "Single Page of Install Instructions".
Thanks. I needed a good instruction manual for the pyTiVo installation. I had tried before to install it but had some issues. These instructions should allow me to get it installed and working properly. Thanks.
I get stuck just a couple steps in. When attempting to install the wmcbrine-2009.03.19-RC1 guide/instructional/etc., it won't continue because it brings up a window, "Checking for Python". It's looking for Python 2.4 or higher, but only 2.4, 2.5 or 2.6. Because Python is now up to 2.7, should I just shuck this old pyTivo installation guide and install pyTivo on my own, or, is there some way to work around this, since the wmcbrine guide hasn't been updated to accept 2.7?
 
#2,443 ·
Please don't call it "the wmcbrine guide"... I'm not responsible for the Windows installer, and my recommendation is that you don't use it at all. You don't need an installer for pyTivo. If you want to run it as a service (which is also not necessary, unlike with TiVo Desktop, although it's available as an option), then follow these instructions (note: also not written by me).
 
#2,444 ·
Sorry, saw that you had stated something similar in another thread, re: the guide.

Unfortunately, I cannot seem to make any real headway with any of the instructional information provided by yourself and others relating to pyTivo installations. Three hours and I've barely made any progress. This stuff is absolutely not for the faint of heart. I'm ready to scream & throw things.

ok ok, I've already screamed & thrown things. :eek:
 
Top