TiVo Community Forum banner
  • TiVoCommunity.com Ambassador Program Now Open! >>> Click Here

New program for 1 step TTG downloads, decryption, encoding - kmttg

2M views 13K replies 921 participants last post by  mattack 
#1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 
See less See more
#2,152 ·
Took my a while to get back...sorry. Just close up this question, UNC paths (\\server\sharename) works perfectly. Time to donate!
Actually, this should not be a surprise. Drive mappings are part of your login session. If you were to log in as a different user, you would have a totally different set of drive mappings, possibly none at all. Since pytivo is running as a service - akin to a unix daemon - it is most certainly NOT running as you and so therefore does not have access to your drive mappings. UNC syntax is the only other possible syntax you could use.
 
#2,153 ·
Seems I'm having a problem with kmttg's qsfix step. I haven't had any processing beyond that step for a couple of days now. I can see VIDEOR~1.EXE in Windows Task Manager sitting at 0% cpu utilization. It just seems to be stuck!

Here's a partial listing of my auto.ini:

Code:
# kmttg auto.ini file
<check_tivos_interval>
30

<dryrun>
0

<dateFilter>
1

<dateOperator>
less than

<dateHours>
72.0

<suggestionsFilter>
1

<kuidFilter>
0

<programIdFilter>
1


<title>
30 Rock
<options>
enabled 1
tivo all
metadata 1
decrypt 1
qsfix 1
twpdelete 0
comskip 1
comcut 1
captions 0
encode 0
push 1
custom 1
suggestionsFilter 0
useProgramId_unique 0
encode_name ff_h264_high_rate
comskipIni none
All s follow the same format, with the exception of an occasional comskipIni override. The last successful "soup to nuts" processing took place on 4/4/2010. Any ideas on what would cause this? :confused: I've bounced the kmttg service in an attempt to get things flowing again, without success.

kmttg version: v0p7j
VideoReDo version: 3.20.1.597 (installed 3/27/2010)
 
#2,154 ·
I'm using kmttg to pull videos from the TiVo, decrypt them into .mpg format, and create corresponding metadata files. I've found that if I play the .mpg back using WMP 11, for some of the videos WMP will crash, saying it got an unexpected termination command. I copied the files to another PC and got the same result. Through trial and error I found that the presence of the metadata file was the cause of the WMP crash - if I remove/rename the metadata file then the video plays without a problem. I compared metadata files between "good" and "bad" videos and can't see a difference. If I copy the contents of a bad metadata file into that of a good one, then it'll cause the "good" video to crash. I edited a "bad" file to see if I could determine the problem, but couldn't turn up anything definite (e.g.,no apparent embedded special characters) other than if I delete enough of the file entries (and it doesn't seem to care which ones) I can get the metadata file to not crash WMP. It's not file length related since I have good metadata files with considerably larger number of characters. I can also "fix" the file by converting the field separators (which appear as squares in Notepad) into paragraph marks (i.e., in Notepad I delete the square and hit "Enter"). Any thoughts on what WMP is finding objectionable in the metadata files?
 
#2,155 ·
Any thoughts on what WMP is finding objectionable in the metadata files?
It's a total surprise to me that WiMP is doing anything with the files -- they aren't intended for it. If the metadata files are showing up in notepad with boxes (LFs) as the line delimiters, and if changing those to CRLFs fixes them, then perhaps kmttg should write them out as native text files. (This is what pyTivo does now in its ToGo plugin, and when reading them it opens them in Universal mode, so the line endings don't matter.) Meanwhile, I'd like to find out more about what WiMP is doing...
 
#2,157 ·
Seems I'm having a problem with kmttg's qsfix step. I haven't had any processing beyond that step for a couple of days now. I can see VIDEOR~1.EXE in Windows Task Manager sitting at 0% cpu utilization. It just seems to be stuck!
auto.log has more relevant information as to what may be going wrong. A few things to try:
1. Make sure qsfix task can run in GUI mode.
2. Check that kmttg service is running with your user account (same account that works when running in GUI mode).
3. Toggle "Allow multiple VideoRedo jobs at once" setting and try again in auto mode.
 
#2,158 ·
It's a total surprise to me that WiMP is doing anything with the files -- they aren't intended for it.
Actually, VLC complains about the metadata files too - although it does not crash - just a simple warning. What's more, VLC seems to have some lattitude about the name of the metadata file - it does not have to be an exact match.
 
#2,159 ·
I'm using kmttg to pull videos from the TiVo, decrypt them into .mpg format, and create corresponding metadata files. I've found that if I play the .mpg back using WMP 11, for some of the videos WMP will crash, saying it got an unexpected termination command. I copied the files to another PC and got the same result. Through trial and error I found that the presence of the metadata file was the cause of the WMP crash - if I remove/rename the metadata file then the video plays without a problem. I compared metadata files between "good" and "bad" videos and can't see a difference. If I copy the contents of a bad metadata file into that of a good one, then it'll cause the "good" video to crash. I edited a "bad" file to see if I could determine the problem, but couldn't turn up anything definite (e.g.,no apparent embedded special characters) other than if I delete enough of the file entries (and it doesn't seem to care which ones) I can get the metadata file to not crash WMP. It's not file length related since I have good metadata files with considerably larger number of characters. I can also "fix" the file by converting the field separators (which appear as squares in Notepad) into paragraph marks (i.e., in Notepad I delete the square and hit "Enter"). Any thoughts on what WMP is finding objectionable in the metadata files?
Confirmed. Not sure what WMP is trying to do with the txt file either, but it appears very picky in wanting \r character at end of line (i.e. Notepad compatible format) in order to prevent a crash.
I suppose I can change metadata file creation to use \r character so that WMP is happy with it.
 
#2,160 ·
Confirmed. Not sure what WMP is trying to do with the txt file either, but it appears very picky in wanting \r character at end of line (i.e. Notepad compatible format) in order to prevent a crash.
I suppose I can change metadata file creation to use \r character so that WMP is happy with it.
WMP is trying to open the .txt file looking for CC info. It accepts either .srt or .txt files.
 
#2,161 ·
auto.log has more relevant information as to what may be going wrong. A few things to try:
1. Make sure qsfix task can run in GUI mode.
2. Check that kmttg service is running with your user account (same account that works when running in GUI mode).
3. Toggle "Allow multiple VideoRedo jobs at once" setting and try again in auto mode.
VRD is popping up a message indicating: "Mpeg stream error: TiVo file open error, please check your media access key." when I use kmttg GUI.

The MAK entered in kmttg is the exact MAK found on tivo.com (actually copy/pasted from the site into kmttg), and is the exact MAK found on my DVRs.

I'm at a loss... The only thing I can think of that changed is unchecking kmttg's option to force TS download when available, due to the problems TS streams cause. Any other suggestions?

Edit: Actually, another change I just discovered was the option to have VRD decrypt instead of TiVodecode. I disabled that option, and will test further...
 
#2,162 ·
Only if you want VideoReDo to understand transport stream .TiVo files. TD 2.6 knows nothing about them; hence, neither does VideoReDo on your system.
Thanks very much. It looks like that did the trick. I take it I missed the memo on that change. (It's too bad that tivodecode isn't quite there yet on TS format, but good to know that the new TD codec does the job.)

TS transfer speeds (TTG) are very impressive indeed, on the new Premiere's. I managed to do a solid 48 MBit/s TTG(ts), while simultaneously doing an MRV from my S3 at 39 MBit/s, for an aggregated total of 87 MBit/s. Running an isolated TTG/ts yields 70+ MBit/s. Awesome!
 
#2,163 ·
VRD is popping up a message indicating: "Mpeg stream error: TiVo file open error, please check your media access key." when I use kmttg GUI.
That implies that TiVoDirectShowFilter (part of TiVo Desktop installation) is not being found properly to decrypt TiVo files. You can check that by trying to play TiVo file with Windows Media Player. If you use tivodecode to decrypt instead then you don't need TiVoDirectShowFilter so can bypass the problem.
 
#2,167 ·
That implies that TiVoDirectShowFilter (part of TiVo Desktop installation) is not being found properly to decrypt TiVo files. You can check that by trying to play TiVo file with Windows Media Player. If you use tivodecode to decrypt instead then you don't need TiVoDirectShowFilter so can bypass the problem.
Thanks K -
It seems I've uninstalled TD+ on that machine. TiVo corp. wouldn't support running their app on Windows Server 2003.

I'm still testing tivodecode & VRD. And I can't win for losing (tonight):

Code:
2010_04_07_01:48:57 >> DECRYPTING F:\FromTivo\V\-Pound of Flesh.TiVo ...
2010_04_07_01:48:57 "C:\Program Files\kmttg\tivodecode\tivodecode.exe" --mak MAK --out "F:\FromTivo\V\-Pound of Flesh.mpg" "F:\FromTivo\V\-Pound of Flesh.TiVo" 
2010_04_07_01:48:59 ERROR: tivodecode failed (exit code: 10 ) - check command: "C:\Program Files\kmttg\tivodecode\tivodecode.exe" --mak MAK --out "F:\FromTivo\V\-Pound of Flesh.mpg" "F:\FromTivo\V\-Pound of Flesh.TiVo" 
2010_04_07_01:48:59 ERROR: processing frame: No error
2010_04_07_01:48:59 ERROR: Encryption by QUALCOMM ;)
2010_04_07_01:48:59 ERROR: Invalid MAK -- aborting
2010_04_07_01:49:00 ERROR: Received unexpected response for: http://localhost:9032/TiVoConnect?Command=Push&Container=Cut+Programs&File=/Lost/-Happily+Ever+After_cut.mpg&tsn=TivoS4+%282%29
2010_04_07_01:49:00 ERROR: Internal Server Error
2010_04_07_01:49:01 >> Running qsfix on F:\FromTivo\V\-Pound of Flesh.mpg ...
2010_04_07_01:49:01 C:\WINDOWS\system32\cscript.exe //nologo "C:\Documents and Settings\orangeboy\Local Settings\Temp\VRD4000477964561124541.vbs" "F:\FromTivo\V\-Pound of Flesh.mpg" "F:\FromTivo\V\-Pound of Flesh.mpg.qsfix" "/l:C:\Documents and Settings\orangeboy\Local Settings\Temp\VRDLock3775289059848572103.tmp" 
2010_04_07_01:49:09 ERROR: qsfix failed (exit code: 0 ) - check command: C:\WINDOWS\system32\cscript.exe //nologo "C:\Documents and Settings\orangeboy\Local Settings\Temp\VRD4000477964561124541.vbs" "F:\FromTivo\V\-Pound of Flesh.mpg" "F:\FromTivo\V\-Pound of Flesh.mpg.qsfix" "/l:C:\Documents and Settings\orangeboy\Local Settings\Temp\VRDLock3775289059848572103.tmp" 
2010_04_07_01:49:09 ERROR: C:\Documents and Settings\orangeboy\Local Settings\Temp\VRD4000477964561124541.vbs(32, 1) (null): The server threw an exception.
I'll look at it more tomorrow.
 
#2,168 ·
Thanks very much. It looks like that did the trick. I take it I missed the memo on that change. (It's too bad that tivodecode isn't quite there yet on TS format, but good to know that the new TD codec does the job.)
Nope. Looks like I jumped the gun. Close, but no cigar. :(

I took a look at all the files I had downloaded previously in TS mode, and then downloaded a few more just to be sure. They all have numerous glitches in them, though certainly far fewer than with tivodecode.

One example, TS glitches at: 5,70,194,302 seconds.
Same program in PS mode: none.
tivodecode would likely be: 3,6,9,12,... seconds.

The glitches are far enough apart, and sometimes subtle enough, that they may not be immediately obvious to everyone. But they're a definite problem. And they may not even be on the PC (decoder) side. It could be a bug in the TTG/ts packager in the TiVo Premieres.

I'm hopeful it is the former, though it could be the later. In any event, ATM all my TS downloads are trash. :down:

I might not mind so much, if it were just a matter of speed. I can live with slower, if I have to. As moyekj suggested, stick with PS mode for now. However, I was hoping to take advantage of the TS mode to download recordings that the TiVo PS mode is still broken on, and truncates midstream. TS mode does this great (never a truncated transfer, out of ~20 tests), but none of the files are any good. Unless you're happy with about a 1 second burst of trash every 1-3 minutes.

The only thing left to try I can think of is downloading via the web interface in TS mode, though I can't imagine that curl being used by kmttg would be a problem. But I'm trying to whittle down the potential problem sources.

What about sending the unmodified .TiVo (via TS) file back up to the Premiere? Would that be worth trying? I.e., if it played then, that would mean the PC-side codec was the problem, and if it didn't, it would isolate the problem to the TTG/ts packager? Any thoughts on that?
 
#2,170 ·
Hmm. Perhaps we should consider changing the extension?
I'd say let MS fix Windows Media Player, but since WMP 12 is already out and only runs on Windows 7, the chances of them going back to fix WMP 11 are nil.

A workaround for this issue on Windows is to open up the metadata file and save it with CR LF instead of LF. I was wondering why I have never seen this issue before and it's because I always manually edit the metadata files, thus changing LFs to CRLFs.

A less intrusive alternative to chaning the file extension for metadata files may be to create and option on the metadata generation to output Windows style metadata files with CRLF instead of LF.
 
#2,171 ·
I know a lot of you guys are using pytivo for goback, but I've just gotten kmttg and galleon going again on a windows box and I'm liking it. I can get basic tagging via file format using kmttg so that Galleon presents the video with Show/Episiode Name correctly. The only thing I'm missing is a process to rename a file with *_cut.mpg" to remove the "_cut" portion. I may have to write a small program to do it if I can't get it done in a batch file.
 
#2,172 ·
A less intrusive alternative to chaning the file extension for metadata files may be to create and option on the metadata generation to output Windows style metadata files with CRLF instead of LF.
This is already checked into SVN for next release. However there really is no excuse from MSFT side to crash an application just because a text file is not what is expected... As mentioned VLC also tries to do something with it and pops up an error message dialog box when it fails and keeps going for playback, which is the expected response.
 
#2,173 ·
I wondering if someone has already solved this. I'd like a custom command script which copies the encoded file, metadata file & .srt file to a specific subdirectory based on the file that was encoded. For example, I'd like episodes of Lost to go into the Lost subdirectory, episodes of House to go into the House subdirectory. I could write a small program to parse the filename or if kmttg had an [MainTitle] option available for the custom command, it could save me some time.

Seems like this might be something that is common enough that someone else might have already solved.
 
#2,174 ·
I wondering if someone has already solved this. I'd like a custom command script which copies the encoded file, metadata file & .srt file to a specific subdirectory based on the file that was encoded. For example, I'd like episodes of Lost to go into the Lost subdirectory, episodes of House to go into the House subdirectory. I could write a small program to parse the filename or if kmttg had an [MainTitle] option available for the custom command, it could save me some time.

Seems like this might be something that is common enough that someone else might have already solved.
Doesn't something like this in kmttg config already do that?
Code:
<tivoFileNameFormat>
[mainTitle][/][EpisodeNumber][-][episodeTitle]
I've got a Series2 that is setup for only manual recordings on a single channel, so I setup file naming to be this:
Code:
<tivoFileNameFormat>
[mainTitle][/][wday][.][year][-][monthNum][-][mday][.][hour][.][min]
 
#2,175 ·
Doesn't something like this in kmttg config already do that?

Code:
<tivoFileNameFormat>
[mainTitle][/][EpisodeNumber][-][episodeTitle]
Perhaps it does and I missed it. I thought the customer command script was limited to a different set of keywords than the filename formatter. Not sure how to handle series where mainTitle is not consistent season to season. Take Survivor for instance.

[Edit] Nevermind - Thinking about mainTitle inconsistency in the program guide, I'm going to have to parse filenames anyway to get files where I ultimately want them archived.
 
#2,176 ·
A less intrusive alternative to chaning the file extension for metadata files
BTW, I don't see this as intrusive. I'd continue to recognize the old names, just only generate the new ones going forward. We have about three programs that use this data, AFAIK, so it shouldn't be hard to coordinate. ".txt" was never a good extension anyway (it's too generic), except in that they actually are text files.
 
Top