TiVo Community Forum banner
  • TiVoCommunity.com Ambassador Program Now Open! >>> Click Here

New program for 1 step TTG downloads, decryption, encoding - kmttg

2M views 13K replies 921 participants last post by  mattack 
#1 ·
EDIT: This program has evolved a lot since this 1st post and now is written in Java and much easier to install than before (very easy on Windows and Mac OSX)... See http://sourceforge.net/projects/kmttg/ for details or visit the end of this thread for more up to date information.

kmttg is a Perl/Tk program I wrote to facilitate TivoToGo (TTG) transfers that can download, create pyTivo metadata, decrypt, run comskip & comcut (commercial detection and removal) and re-encode multiple shows you select from your Tivos all in 1 step.
You can select one or more shows at a time and then with one click of a button the program will download all the selected items, with the options of also automatically creating a metadata file for pyTivo, decrypting .TiVo files to .mpg, running comskip (commercial detection and removal program), and automatically re-encoding to a more portable format using mencoder, ffmpeg or any other command line encoder of your choosing. The program queues up multiple jobs and displays time, size and speed statistics for ongoing jobs.

For more information, screenshots and download visit:
http://sourceforge.net/projects/kmttg

Web page contains windows executables for all 3rd party tools used. The only other requirement of course is you must have Perl installed. Runs under Windows or Linux - tested with WinXP SP2 & Linux Red Hat Enterprise 4.

If you do try this out would appreciate some comments/feedback. For any programmers out there feel free to tinker and make improvements yourself.
 
See less See more
#2,995 ·
It may be easier for you to install and run the Handbrake GUI so that you can experiment with different settings (use a small source clip as a test to speed things up). Then once you find something that works/you are happy with you can have the GUI show you what the command line equivalent is from which it is easy to generate your own handbrake profile for kmttg.
Thanks - I have the handbrake GUI but do not see where you can get the command line equivalent
I checked the mpg decoded file from TiVo loaded in HB:

TiVo Source Device Model 649080
352x480 Output 352x480 Anamorphic 308x480 Loose
(which to me already looks wrong...?)

I checked my other 2 TiVos and looked these mpg encoded files look fien in HB:
TiVo Source Device Model 240040
352 x 480 Output 352x480 Anamorphic 639x480 Loose (looks OK)
TiVo Source Device Model 540040
352 x 480 Output 352x480 Anamorphic 639x496 Loose (looks OK)

Files are recorded at same resolution but on different Tivo's.
The one with problem is a Dual Tuner device 649080
The other two older TiVos Model 240040 and 540040 look right

All files are transferred & decrypted by kmttg

THe problem seems to be in the source files not in the kmttg but I cannot understand why to where this would be set/changed?
 
#2,996 ·
If the goal is just to play back on TiVo then leaving it as mpeg2 is the best option. Many people like to try and re-encode to reduce file size but to me it's not worth the time, effort and quality degradation to do so if playback device supports the original recording format. I do take time to manually strip out commercials with VRD if I intend to take a bunch of shows on the road on my laptop for playback, but I still leave them as mpeg2.
moyekj: If you leave it as mpeg2 for TiVo playback (no commercial stripping), do you still qsfix it?
 
#2,997 ·
Thanks - I have the handbrake GUI but do not see where you can get the command line equivalent
It is hidden by default but can be enabled in the program options menu. Once enabled and you restart Handbrake you will see a "Query Editor" tab where you can query the command line equivalent of your current GUI choices.
 
#2,999 ·
It is hidden by default but can be enabled in the program options menu. Once enabled and you restart Handbrake you will see a "Query Editor" tab where you can query the command line equivalent of your current GUI choices.
I think this may only be in the Windows version
https://trac.handbrake.fr/wiki/MacGuiGuide makes no reference to it
It is mentioned in the Windows GUI Page (at the bottom in notes)
https://trac.handbrake.fr/wiki/WindowsGuiGuide

But I do not think the problem is HB - the input file is 352x480 Anamorphic 308x480
This Anamorphic format looks narrow and tall
But for other Tivo files
352 x 480 Output 352x480 Anamorphic 639x480 Loose
This Anamorphic format looks correct

I tried transferring the show from the Tivo that produces this narrow tall output to one of my other Tivos and then re-downloading with kmttg but the narrow tall problem persist.
So it boils down to the DualTUner Tivo producing files that are downloaded that create narrow tall output
Any thoughts greatly appreciated

n
 
#3,000 ·
nvt, it would be helpful if you can post mediainfo output for both for a source file that has issue as well as for one that doesn't. I know you listed resolution information but other info such as aspect ratio is also needed. Also if you know it post the native resolution and aspect ratio of the iPad.
 
#3,001 ·
nvt, it would be helpful if you can post mediainfo output for both for a source file that has issue as well as for one that doesn't. I know you listed resolution information but other info such as aspect ratio is also needed. Also if you know it post the native resolution and aspect ratio of the iPad.
As far as I know:
iPad screen is 1024x768 which if I understand correctly is 4:3

Source file that has problems:
No Ordinary Family - No Ordinary Detention__(01_18_2011).mpg
Format : MPEG-PS
File size : 1.16 GiB
Duration : 59mn 59s
Overall bit rate : 2 770 Kbps
Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Duration : 59mn 59s
Bit rate mode : Variable
Bit rate : 2 523 Kbps
Nominal bit rate : 9 000 Kbps
Width : 352 pixels
Height : 480 pixels
Display aspect ratio : 0.652
Frame rate : 29.970 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.498
Stream size : 1.06 GiB (91%)

Audio
ID : 192 (0xC0)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Duration : 59mn 59s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : 11ms
Stream size : 82.4 MiB (7%)

Text
ID : 224 (0xE0)-608-1
Format : EIA-608
Muxing mode : A/53 / DTVCC Transport
Muxing mode, more info : Muxed in Video #1
Stream size : 0.00 Byte (0%)

This is a file that came from a different Tivo that encodes fine using hb_appletv:

Tosh.0 - The National Anthem girl_304_(02_01_2011).mpg
Format : MPEG-PS
File size : 662 MiB
Duration : 35mn 0s
Overall bit rate : 2 643 Kbps

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=15
Duration : 35mn 0s
Bit rate mode : Variable
Bit rate : 2 399 Kbps
Nominal bit rate : 2 600 Kbps
Width : 352 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 29.970 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.474
Stream size : 601 MiB (91%)

Audio
ID : 192 (0xC0)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Duration : 35mn 0s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : 17ms
Stream size : 48.1 MiB (7%)

Text
ID : 224 (0xE0)-608-1
Format : EIA-608
Muxing mode : A/53 / DTVCC Transport
Muxing mode, more info : Muxed in Video #1
Stream size : 0.00 Byte (0%)

Thanks
 
#3,002 ·
The "bad" source has a very strange aspect ratio which is the root of the problem:
Display aspect ratio : 0.652

The "good" source has conventional aspect ratio:
Display aspect ratio : 4:3

I'm not sure what Handbrake does in that situation but my guess is it's keeping the same strange aspect ratio which is why it displays funny on your iPad. You should also run the output mp4 file through mediainfo to see the resulting encoding specs. It may be possible to fix things with some cropping and aspect ratio specifications in Handbrake, but sure is strange looking.
 
#3,003 ·
The "bad" source has a very strange aspect ratio which is the root of the problem:
Display aspect ratio : 0.652
The "good" source has conventional aspect ratio:
Display aspect ratio : 4:3
I'm not sure what Handbrake does in that situation but my guess is it's keeping the same strange aspect ratio which is why it displays funny on your iPad. You should also run the output mp4 file through mediainfo to see the resulting encoding specs. It may be possible to fix things with some cropping and aspect ratio specifications in Handbrake, but sure is strange looking.
Agreed but I am at a loss to explain why that would be the case. I will review the TiVo in question but as far as I know there is no difference in settings and I am not aware of anything in Tivo that specifies this for recording?
I am going to chase with Tivo support to see if they can shed any light on it and I'll report back
Thanks for your help
 
#3,004 ·
The "bad" source has a very strange aspect ratio which is the root of the problem:
Display aspect ratio : 0.652
The "good" source has conventional aspect ratio:
Display aspect ratio : 4:3
I'm not sure what Handbrake does in that situation but my guess is it's keeping the same strange aspect ratio which is why it displays funny on your iPad. You should also run the output mp4 file through mediainfo to see the resulting encoding specs. It may be possible to fix things with some cropping and aspect ratio specifications in Handbrake, but sure is strange looking.
Here's the update - spoke with Tivo support who attribute this to a player codec problem...!

What was interesting though when I was testing playing the file that has the the odd aspect ratio in VLC it plays fine and looks like but MediaInfo for this same file reports Display aspect ratio: 0.652

So the file plays OK in a media player (VLC) but when converted turns into a narrow tall playback with the following media inspector output

No Ordinary Family - No Ordinary Brother__(01_11_2011) HB_appletv.m4v
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42
File size : 272 MiB
Duration : 41mn 10s
Overall bit rate : 925 Kbps
Movie name : No Ordinary Brother
Part/Position : 1
Part/Total : 1
Performer : No Ordinary Family
Genre : Drama
Encoded date : UTC 2011-01-11 00:00:00
Tagged date : UTC 2011-02-02 13:05:39
Writing application : HandBrake 0.9.4 2009112300
stik : 10
desc : Jims brother learns of the familys powers and tries to use it to his advantage; Katie is promoted; Daphne becomes a part of the schools peer disciplinary committee.
tvsh : No Ordinary Family
tvnn : WJLA

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L2.1
Format settings, CABAC : No
Format settings, ReFrames : 2 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 41mn 10s
Bit rate mode : Variable
Bit rate : 760 Kbps
Width : 352 pixels
Height : 480 pixels
Display aspect ratio : 0.642
Frame rate mode : Constant
Frame rate : 29.970 fps
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.150
Stream size : 224 MiB (82%)
Writing library : x264 core 79 r1347 5ddd61b
Encoding settings : cabac=0 / ref=2 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=7 / psy=1 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=6 / nr=0 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=1 / wpredb=0 / wpredp=0 / keyint=300 / keyint_min=30 / scenecut=40 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2011-02-02 12:46:30
Tagged date : UTC 2011-02-02 13:05:39
Color primaries : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 41mn 9s
Bit rate mode : Variable
Bit rate : 160 Kbps
Maximum bit rate : 229 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Stream size : 47.1 MiB (17%)
Encoded date : UTC 2011-02-02 12:46:30
Tagged date : UTC 2011-02-02 13:05:39

Text
ID : 3
Format : Apple text
Codec ID : text
Duration : 41mn 10s
Bit rate : 0
Stream size : 14.0 Bytes (0%)
Encoded date : UTC 2011-02-02 12:46:30
Tagged date : UTC 2011-02-02 13:05:39

Now I am very confused.....
 
#3,005 ·
The resulting m4v file by Handbrake also looks OK with VLC player right? If so then TiVo support is not wrong in saying that the iPad player just can't handle proper display of that aspect ratio. I believe I saw a post somewhere saying that you can install different media player apps on the iPad (rather than using the built in player), so that may be one possible solution.
 
#3,006 ·
Agreed but I am at a loss to explain why that would be the case. I will review the TiVo in question but as far as I know there is no difference in settings and I am not aware of anything in Tivo that specifies this for recording?
I am going to chase with Tivo support to see if they can shed any light on it and I'll report back
Thanks for your help
I had a similar issue with a TiVo file and HB a couple of years ago. The gory details may be found here:

https://forum.handbrake.fr/viewtopic.php?f=11&t=8540

The signaling of DAR in mpeg2 files is somewhat complicated, as you will see if you can stay awake through that thread.

The bottom line seemed to be:
1. HB was following the MPEG2 spec.
2. The TiVo file was not to spec.
3. But VRD, gspot, mediaInfo and VLC have their own ways of handling DAR so they sometimescan give a desirable result even though they may not be rigorously following the MPEG2 spec.

I think the explanation probably is that the two models of TiVo differ in how they incorporate the aspect ratio signaling into the file.

I ended up slightly modifying and building the Windows version of the HB command line program so it would work the way I wanted. On the PC I had at the time this build took over an hour. HB has gone through 2 new versions since then so all this is obsolete now. Hopefully there will be some command line option combo that will handle the problem for you.
 
#3,008 ·
The resulting m4v file by Handbrake also looks OK with VLC player right?
In fact it is not - the file once converted changes to the narrow/tall aspect ratio. VLC Info says 352x480.
In the sequence of files created;
TiVo (the direct download still encrypted) -
mpg decode - VLC plays this correctly
cut - VLC plays correctly
But any of the conversions using HB play incorrectly in VLC (HB_appletv, HB_film, HB_ps3, HB_television)
But conversion using ff_ipad plays correctly (but the reason I stopped using ff was it de synched the audio.

I interpret this as Handbrake failing to convert correctly and I end up with sound playing disconnected from the video.

I believe I saw a post somewhere saying that you can install different media player apps on the iPad (rather than using the built in player), so that may be one possible solution.
VLC does make a player but it is unstable and hangs every 10-15 minutes on the iPad making it unusable

However the post from dlfl it seems the problem occurs in the difference between HB interpretation of the stream

I think the explanation probably is that the two models of TiVo differ in how they incorporate the aspect ratio signaling into the file.
I ended up slightly modifying and building the Windows version of the HB command line program so it would work the way I wanted. On the PC I had at the time this build took over an hour. HB has gone through 2 new versions since then so all this is obsolete now. Hopefully there will be some command line option combo that will handle the problem for you.
So I tested some manual encoding of the "cut" encoded profile using handbrake and was able to get it to format the show into a reasonable output
THis was the specs retrieved from the activity window
job configuration:
+ container: MPEG-4 (.mp4 and .m4v)
+ decoder: mpeg2
+ bitrate 9000 kbps
+ frame rate: 29.970 fps -> peak rate limited to 29.970 fps
+ custom anamorphic
+ storage dimensions: 352 * 480 -> 352 * 480, crop 2/0/4/2, mod 16
+ pixel aspect ratio: 20 / 11
+ display dimensions: 640 * 480
+ encoder: x264
+ quality: 20.00 (RF)
* audio track 0
+ decoder: Unknown (MPEG) (2.0 ch) (track 1, id c0)
+ mixdown: Stereo
+ encoder: ca_aac
+ bitrate: 160 kbps, samplerate: 48000 Hz
reader: first SCR 819 id 224 DTS 18000
encx264: min-keyint: auto (29), keyint: 300
encx264: encoding with stored aspect 20/11
encx264: Encoding at constant RF 20.000000
x264 [info]: using SAR=20/11
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2

This was using Handbrake Mac GUI v0.9.5
kmttg has Handbrake CLI v 0.9.4 although I could download 0.9.5 direct from the handbrake site (CLI Interface: http://handbrake.fr/downloads2.php
I used the CLI help page https://trac.handbrake.fr/wiki/CLIGuide and put together an enc profile (hb_ipadcustom) that used some of the iPad presents on this page:
https://trac.handbrake.fr/wiki/BuiltInPresets

My attempt
# Description (single line, keep short)
<description>
Handbrake ipadcustom: m4v container, AppleTV preset (h.264 video, aac & ac3 audio)
# Encode command or script (single line)
# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
<command>
HANDBRAKE -i INPUT --cpu CPU_CORES
-e x264 -q 20.0 -r 29.97 --pfr --crop 2:0:4:2 --custom-anamorphic --display-width 640 --pixel-aspect 20:11 --modulus 16 -a 1 -E faac -B 160 -6 dpl2 -R Auto -D 1,1 -f m4v -4 -X 1024 -m -v -o OUTPUT
# Encoded output file extension
<extension>
m4v

I will try this but any thoughts/input on the specs would be appreciated.
 
#3,009 ·
I've been going crazy today trying multiple configurations but have failed to get the right config in the encode profile.

I can get the aspect ration working but the HB conversion ends up creating a constant "judder".
I've tried the apple_tv preset and modified settings in that (based on the hb_appletv.enc" profile.
I've tried using the iPad preset from HB's CLI preset web page and combinations
But the judder in the video persist and I am not sure what setting to try changing to make this work.
If I do the translation manually in Handbrake it has no judder

My latest settings
HANDBRAKE -i INPUT --cpu CPU_CORES -e x264 -q 20.0 -r 29.97 --pfr --crop 2:0:4:2 --custom-anamorphic --pixel-aspect 20:11 --modulus 16 -a 1 -E faac -B 160 -6 dpl2 -R Auto -D 1,1 -f m4v -4 -X 1024 -m -v -o OUTPUT

If anyone has better insight into settings I would greatly appreciate some help

Thanks,
 
#3,010 ·
I've been going crazy today trying multiple configurations but have failed to get the right config in the encode profile.

I can get the aspect ration working but the HB conversion ends up creating a constant "judder".
I've tried the apple_tv preset and modified settings in that (based on the hb_appletv.enc" profile.
I've tried using the iPad preset from HB's CLI preset web page and combinations
But the judder in the video persist and I am not sure what setting to try changing to make this work.
If I do the translation manually in Handbrake it has no judder

My latest settings
HANDBRAKE -i INPUT --cpu CPU_CORES -e x264 -q 20.0 -r 29.97 --pfr --crop 2:0:4:2 --custom-anamorphic --pixel-aspect 20:11 --modulus 16 -a 1 -E faac -B 160 -6 dpl2 -R Auto -D 1,1 -f m4v -4 -X 1024 -m -v -o OUTPUT

If anyone has better insight into settings I would greatly appreciate some help

Thanks,
I think you are getting judder because you are not detelecining and are forcing the framerate to 29.97. Try enabling detelecine (and I would do decomb too) and set the fps to 'Same as Source'. That should give you a progressive stream at 23.976fps (normally, sometimes video will have the interlaced scenes in the credits). What I would expect HB to do if told to output 29.97fps would be to leave the telecined frames in the stream rather than re-build the progressive frames. That is the source of your judder.
 
#3,011 ·
I think you are getting judder because you are not detelecining and are forcing the framerate to 29.97. Try enabling detelecine (and I would do decomb too) and set the fps to 'Same as Source'. That should give you a progressive stream at 23.976fps (normally, sometimes video will have the interlaced scenes in the credits). What I would expect HB to do if told to output 29.97fps would be to leave the telecined frames in the stream rather than re-build the progressive frames. That is the source of your judder.
Thxs Txporter....in the meantime one of my attempts worked:
HANDBRAKE -i INPUT --cpu CPU_CORES --preset iPad --custom-anamorphic --display-width 640 -f m4v -D 1,1 -A "Stereo","AC3" --rate 29.97 -v -o OUTPUT

I'll try and build a new one based on your suggestions but post the above just in case anyone searches for a solution
Thanks to all for help - greatly appreciated. I now know a lot more about video formats than I did a few days ago
 
#3,013 ·
Is the video filter you mentioned, the video dimension filter (enabled in kmttg)?
Yes, it's the kmttg configuration option in VideoRedo tab:
Enable VideoRedo QS Fix video dimension filter

VRD doesn't like editing videos that contain more than 1 resolution of video (error message and/or crash when editing). For many cable companies when they splice in commercials this can result in video with different resolutions than the program. For example for my cable company FNCHD programming is 720p but often there are commercials that are 1080i and hence without using video filter I cannot edit out commercials using VRD.
 
#3,015 ·
I'm getting roughly 30 Mbps through a wired ethernet connection on my Premiere using kmttg. I'm running through a router. Any idea how I could possibly increase these transfer speeds? I'd like to get 60 Mbps.
Tune both tuners to music channels or channels you don't subscribe to. Still won't get you 60 but should get you higher than 30.
 
#3,017 ·
moyekj,
I don't quite follow you. How would that improve my transfers of recorded shows to my PC? Maybe I didn't clarify it well enough in my previous post. Thanks.
Transferring shows to your PC from TiVo is a CPU and resource intensive task (decryption, demuxing, re-muxing, re-encryption). By tuning to channels you don't receive or music channels it helps free up resources on TiVo and thus speeds up transfers quite significantly - try it and you should see a speedup, though probably not as significant as you would like.
 
Top