TiVo Community Forum banner
  • TiVoCommunity.com Ambassador Program Now Open! >>> Click Here

New program for 1 step TTG downloads, decryption, encoding - kmttg

2M views 13K replies 921 participants last post by  mattack 
#1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 
See less See more
#4,270 ·
Did that, but pyTiVo doesn't seem to be parsing it in the first place. Do you also have to modify metadata.py? I'd like for pyTiVo to parse it from metadata text files as well as from .tivo files.
I thought that the mod to metadata.py which I made was only required to get the programId into the text file but I could be wrong. There may also be a mod to another module, I don't remember exactly what I did and I won't be able to check until at least Tuesday.

Is the programId not being sent to the TiVo?
 
#4,271 ·
lpwcomp, I already have the necessary updates checked into source if you want to review or test it out (for both metadata from shows still on TiVo as well as from .TiVo files). Obtaining from .TiVo was a little tricky as "uniqueId" is used for both seriesId and programId in the tivodecode xml dump but from my short testing I think it's working there as well.
For .tivo metadata extraction, shouldn't you be looking at chunk 2, not 1. i.e. "chunk-02-0002.xml", not "chunk-01-0001.xml"? Do a test dump w/tivodecode and compare the 2 chunks! Also the seriesId and programId both need to be extracted from <showing>, not <vActualShowing>. If either gets pulled from <vActualShowing> and the recording was padded, the value will be incorrect for the recording.
 
#4,273 ·
I thought that the mod to metadata.py which I made was only required to get the programId into the text file but I could be wrong. There may also be a mod to another module, I don't remember exactly what I did and I won't be able to check until at least Tuesday.
What mod? What fork? I'm running wmcbrine's 2012-07-11 with a hand edited TvBus.tmpl to add programId. Querying pyTiVo, it doesn't show up in the Container or TVBus video details queries.
 
#4,274 ·
For .tivo metadata extraction, shouldn't you be looking at chunk 2, not 1. i.e. "chunk-02-0002.xml", not "chunk-01-0001.xml"? Do a test dump w/tivodecode and compare the 2 chunks! Also the seriesId and programId both need to be extracted from <showing>, not <vActualShowing>. If either gets pulled from <vActualShowing> and the recording was padded, the value will be incorrect for the recording.
It is parsing chunk 2 and from <showing> section. From metadataTivo.java:
Code:
metaFileFromXmlFile(xmlFile2, job.metaFile);
(and xmlFile2 is chunk 2)

and under metaFileFromXmlFile method:
Code:
         // Search for everything under <showing>
         NodeList nlist = doc.getElementsByTagName("showing");
...
 
#4,275 ·
What mod? What fork? I'm running wmcbrine's 2012-07-11 with a hand edited TvBus.tmpl to add programId. Querying pyTiVo, it doesn't show up in the Container or TVBus video details queries.
There is no fork with my mods and I never meant to imply there was. I made the mods to my own copy of pyTivo, just as I am running a self-modified version of kmttg.

As far as getting it to the TiVo is concerned, the fact that even if it does get to the TiVo, it doesn't always do what I expected, i.e. set the Season and Episode, probably led me to lose track of exactly what I did to get it to be transmitted. Well, that plus I was also trying to get grouping on a Premiere of pulled programs with an inactive or arbitrary seriesId to work again.
 
#4,276 ·
It is parsing chunk 2 and from <showing> section. From metadataTivo.java:
Code:
metaFileFromXmlFile(xmlFile2, job.metaFile);
(and xmlFile2 is chunk 2)

and under metaFileFromXmlFile method:
Code:
         // Search for everything under <showing>
         NodeList nlist = doc.getElementsByTagName("showing");
...
That's why you're doing the coding and not me! My cursory glance at the code missed this part
Code:
xmlFile2 = xmlFile.replaceAll("1", "2");
 
#4,279 ·
@ moyekj Thanks!
@ moyekj or any one else :)
Does anyone know if pytivo is sending or if the tivo is honoring episode number when a show is pushed? I tend to edit and encode to mp4 before pushing back to the tivo, a S3 HDXL, using the metadata generated by kmttg. Now that episode number is available on kmttg I have noticed that pushes do not display the episode number.
Thanks,
OOOOPS
 
#4,280 ·
@ moyekj Thanks!
@ moyekj or any one else :)
Does anyone know if pytivo is sending or if the tivo is honoring episode number when a show is pushed? I tend to edit and encode to mp4 before pushing back to the tivo, a S3 HDXL, using the metadata generated by kmttg. Now that episode number is available on kmttg I have noticed that pushes do not display the episode number.
Thanks,
OOOOPS
Pushes send very little of the metadata. episodeNumber has been part of the metadata since the beginning, the change to kmttg was just to display it as part of the title in the gui.
 
#4,281 ·
It's not.
Just tested. I started with a fresh install of pyTiVo, which naturally didn't send the programId that was in the metadata file. I then edited pyTivo/plugins/video/templates/TvBus.tmpl and added:
Code:
      #if $video.programId
      <uniqueId>$video.programId</uniqueId>
      #end if
right before
Code:
</program>
. I then pulled the same recording to the TiVo again. Season and Episode numbers were displayed in the HDUI.

Note that
  • this will not work on a Push since a Push sends very little of the metadata.
  • even if it gets to the TiVo, there is no guarantee that it will do anything.
 
#4,282 ·
Just tested. I started with a fresh install of pyTiVo, which naturally didn't send the programId that was in the metadata file. I then edited pyTivo/plugins/video/templates/TvBus.tmpl and added:
Code:
      #if $video.programId
      <uniqueId>$video.programId</uniqueId>
      #end if
right before
Code:
</program>
. I then pulled the same recording to the TiVo again. Season and Episode numbers were displayed in the HDUI.

Note that
  • this will not work on a Push since a Push sends very little of the metadata.
  • even if it gets to the TiVo, there is no guarantee that it will do anything.
Not sure if that proves anything. Did you look at the container or video details view provided by pyTiVo to your TiVo? What type of metadata and video file. Was the original recording still on your TiVo, etc. The TVBus template provides the video details view, but if that metadata hasn't been parsed by pyTiVo and made available to display, it won't. Haven't seen anything since July from mcbrine on the pyTiVo forum thread. Is it still being maintained?

AND P.S. Why are we discussing this on the kmttg forum thread and not the pyTiVo forum thread?
 
#4,283 ·
Not sure if that proves anything. Did you look at the container or video details view provided by pyTiVo to your TiVo? What type of metadata and video file. Was the original recording still on your TiVo, etc. The TVBus template provides the video details view, but if that metadata hasn't been parsed by pyTiVo and made available to display, it won't. Haven't seen anything since July from mcbrine on the pyTiVo forum thread. Is it still being maintained?

AND P.S. Why are we discussing this on the kmttg forum thread and not the pyTiVo forum thread?
You're right, this discussion belongs in the other forum. However, since you think you know everything about it anyway, you obviously don't need my help. Good luck.
 
#4,284 ·
v0p8u version just released with some fairly minor updates including showing EpisodeNumber as part of title in various tables if available. See release_notes Wiki for details.
YEAH for this change. I know it has been something debated for awhile for all the reasons debated about, but glad to have it as it will really help for those times when it is a. actually there and b. actually correct. which has nothing to do with kmttg obviously as for some reason tivo guide cannot get correct what all kinds of other free software can.

Thanks so much!
 
#4,285 ·
You're right, this discussion belongs in the other forum. However, since you think you know everything about it anyway, you obviously don't need my help. Good luck.
I agree it should be continued in the pyTiVo forum.

Never said I know everything about it, or meant to imply it. If I knew everything, I wouldn't be posting in these forums. I was just providing a somewhat more scientific approach to the problem versus an emperical evidence approach.

We're both trying to achieve the same objective, I was just offering what I knew based on poring over code I'm unfamiliar with and trying to analyze what it's doing. I'm a developer from a previous life and don't want to learn YAPL (yet another programming language) just to get this accomplished.

It appears the mcbrine pyTiVo thread is on-hold or dead. Don't know if it's still being developed and if not, what fork to follow.

Was just trying to offer what I found, but if the attitude is nobody wants to know, I'll STFU and keep it to myself.

Good Luck also.
 
#4,288 ·
There are a large number of movies where the remake has the same name as the original such as Bad Company and Freaky Friday. Any chance as a future update that the movie year could be an option for creating the file name? By the way this program works great - highly recommended. I use it both on my linux box and on my windows PC.
 
#4,289 ·
Anyone done any comparisons? Auto mode without reviewing the cuts. Obviously I'd rather wind up with a stray commercial rather then cut part of the program.

Most network shows have a logo in the corner during the program but not during the commercial. Surprised there isn't a reliable way to use that.
 
#4,290 ·
Anyone done any comparisons? Auto mode without reviewing the cuts. Obviously I'd rather wind up with a stray commercial rather then cut part of the program.

Most network shows have a logo in the corner during the program but not during the commercial. Surprised there isn't a reliable way to use that.
VideoReDo does not determine commercial breaks by the logo on the video, it determines by the black frames between the video and the breaks. Sometimes they will cut directly to the commercial suddenly without the blank frames and that will not get detected.
 
#4,291 ·
VideoReDo does not determine commercial breaks by the logo on the video, it determines by the black frames between the video and the breaks. Sometimes they will cut directly to the commercial suddenly without the blank frames and that will not get detected.
My post wasn't clear. My understanding is comskip can use the logo and VRD doesn't. Hence the question which approach gets better results.
 
#4,292 ·
Most network shows have a logo in the corner during the program but not during the commercial. Surprised there isn't a reliable way to use that.
I never much cared for auto-commercial cutting personally. I'd rather just do it manually since I don't do it very often. But comskip is highly configurable - by editing comskip.ini you can do things such as you suggest to target a network logo as an indicator. But you have to put in place different configs for different channels and/or shows so if you have a wide variety of recordings you are targeting it quickly becomes unwieldy. If you want to play with it, kmttg does allow you to specify comskip.ini file to use and in auto transfers config you can even setup a different comskip.ini for each entry so you could setup different ones for different shows. It all depends how much work you are willing to put in to get it working as well as possible. If it's just occasional use then I say don't bother and just cut manually using VRD without any commercial detection. If you do a lot of shows frequently then it's a different story.
 
#4,293 ·
I never much cared for auto-commercial cutting personally. I'd rather just do it manually since I don't do it very often. But comskip is highly configurable - by editing comskip.ini you can do things such as you suggest to target a network logo as an indicator. But you have to put in place different configs for different channels and/or shows so if you have a wide variety of recordings you are targeting it quickly becomes unwieldy. If you want to play with it, kmttg does allow you to specify comskip.ini file to use and in auto transfers config you can even setup a different comskip.ini for each entry so you could setup different ones for different shows. It all depends how much work you are willing to put in to get it working as well as possible. If it's just occasional use then I say don't bother and just cut manually using VRD without any commercial detection. If you do a lot of shows frequently then it's a different story.
ITA I was thinking of automating the process so I can D/L programs while I'm on vacation. My goal would be to get rid of some, not all, commercials. I have no doubt getting rid of all the commercials is likely to also get rid of some of the program.

I may try it with earlier episodes of programs. See which programs work.

I'll be watching the program when I don't have access to the internet.

This "exercise" may be more trouble then it's worth.
 
#4,294 ·
If it's just occasional use then I say don't bother and just cut manually using VRD without any commercial detection.
I definitely agree with the rest of what you say, but definitely not this. No commercial cutting utility is going to be 100% accurate, or indeed anywhere close, but that does not mean automated commercial cutting is not useful. Having a cut file created automatically can save the user a huge amount of time when manually editing the file. Most cuts are at least in the ballpark, and skipping to the proposed cut points can save one easily ten to fifteen minutes of searching per show on average. False positives are easy to detect and eliminate, and it is usually easy to tell when a commercial has been missed because the show content will be more than 43 minutes per hour of recording after one is at the end of the file. I manually edit all programs - even the ones which do not contain commercials, which in my case is the vast majority - but I have kmttg run them all through comskip when they are downloaded.
 
Top