New program for 1 step TTG downloads, decryption, encoding - kmttg

[windracer]

That's what I figured, thanks.

edit: this worked: I piped the output of my 'kmttg -b' command to a log file, then ran that through sed with the following command:

sed -f /home/jeremy/scripts/kmttg-filter.sed < /var/log/kmttg.log

where the contents of kmttg-filter.sed are:

/curl/d
/>> Continuing/d
/Skipping/d
/job=playlist/d
/NPL job completed/d
/ffmpeg/d
/NOTE:/d

The result is a lot more readable in my daily e-mail.

STARTING BATCH MODE SINGLE LOOP TRANSFERS
>> Getting Now Playing List from Office ...
>> Getting Now Playing List from Family Room ...
TOTAL auto matches for 'Family Room' = 0/236
Title keyword match: 'the simpsons' found in 'the simpsons'
START PROCESSING OF ENTRY: The Simpsons - A Totally Fun Thing Bart Will Never Do Again
TOTAL auto matches for 'Office' = 1/249
>> DOWNLOADING/DECRYPTING TO /etc/tivo/togo/downloads/The Simpsons - A Totally Fun Thing Bart Will Never Do Again (2012-04-29).mpg ...
/etc/tivo/togo/downloads/The Simpsons - A Totally Fun Thing Bart Will Never Do Again (2012-04-29).mpg: size=3109.71 MB elapsed=0:44:53 (9.69 Mbps)
---DONE--- job=download_decrypt output=/etc/tivo/togo/downloads/The Simpsons - A Totally Fun Thing Bart Will Never Do Again (2012-04-29).mpg
>> ENCODING WITH PROFILE 'airvideo_720' TO FILE /etc/tivo/togo/The Simpsons - A Totally Fun Thing Bart Will Never Do Again (2012-04-29).m4v ...
encoding job completed: 1:10:31
---DONE--- job=encode output=/etc/tivo/togo/The Simpsons - A Totally Fun Thing Bart Will Never Do Again (2012-04-29).m4v

EXITING BATCH MODE
SHUTTING DOWN

[nahtanoj1]

Does it make sense that when KMTTG founds files that are configured to auto transfer, that it launches a second version of KMTTG so I actually have two versions of KMTTG running? If I launch the app and it shows me my now playing list do I need to relabel auto transfers each time I launch it (by selecting auto transfers->background job->enable

This is the output from when I choose Auto Transfers->Background job-> Status
Process running: 502 71697 71487 0 9:26PM ?? 0:14.71 /usr/bin/java -jar /Users/jonathanbaker/Applications/kmttg_v0p8r/kmttg.jar -a &

Thanks.

[moyekj]

Originally Posted by nahtanoj1

Does it make sense that when KMTTG founds files that are configured to auto transfer, that it launches a second version of KMTTG so I actually have two versions of KMTTG running? If I launch the app and it shows me my now playing list do I need to relabel auto transfers each time I launch it (by selecting auto transfers->background job->enable

This is the output from when I choose Auto Transfers->Background job-> Status
Process running: 502 71697 71487 0 9:26PM ?? 0:14.71 /usr/bin/java -jar /Users/jonathanbaker/Applications/kmttg_v0p8r/kmttg.jar -a &

Thanks.

You can run auto transfer mode via the GUI instead of as a background process. i.e. Auto Transfers->Loop in GUI if you want it to keep going continuously or Auto Transfers->Run Once in GUI for one pass.

[nahtanoj1]

Originally Posted by moyekj

You can run auto transfer mode via the GUI instead of as a background process. i.e. Auto Transfers->Loop in GUI if you want it to keep going continuously or Auto Transfers->Run Once in GUI for one pass.

Thanks. That will do it.

[SugarBowl]

Just getting started with kmttg and it is working great. Excellent program!

I am downloading from tivo, encoding for appletv, atomicparsley, then a custom command to move the final file into the Automcatically Add to Itunes Folder.

If i have 5 shows that are queued up, each show progress thru the encode step. After the encode step finishes, the atomic and custom step remain queued, and the next show starts transferring and begins encoding.

Every show has to get encoded before any of the atomic and custom commands are run. Is this normal behaviour ?

[moyekj]

Originally Posted by SugarBowl

Just getting started with kmttg and it is working great. Excellent program!

I am downloading from tivo, encoding for appletv, atomicparsley, then a custom command to move the final file into the Automcatically Add to Itunes Folder.

If i have 5 shows that are queued up, each show progress thru the encode step. After the encode step finishes, the atomic and custom step remain queued, and the next show starts transferring and begins encoding.

Every show has to get encoded before any of the atomic and custom commands are run. Is this normal behaviour ?

Under configure->Program Options if you want to allow more jobs to run in parallel then increase the "active job limit" setting.

[SugarBowl]

Originally Posted by moyekj

Under configure->Program Options if you want to allow more jobs to run in parallel then increase the "active job limit" setting.

I increased this from 1 to 2, it will now process 2 jobs at the same time, but the job still skips the atomic and custom steps, and goes onto the next job.

It looks like the 'atomic' step is getting created dynamically, following the completion of the encode step. On the UI, the atomic step is in the expected location within the job, but maybe internally, this step is being added to the bottom of the overall queue?

[moyekj]

Originally Posted by SugarBowl

I increased this from 1 to 2, it will now process 2 jobs at the same time, but the job still skips the atomic and custom steps, and goes onto the next job.

It looks like the 'atomic' step is getting created dynamically, following the completion of the encode step. On the UI, the atomic step is in the expected location within the job, but maybe internally, this step is being added to the bottom of the overall queue?

atomic job is only scheduled when encoding completes so other "active" jobs currently running will get priority or be higher in the queue. Obviously if atomic job is queued up the custom job won't run until after it completes. "active" job is basically anything other than a download task. If you really want to force some queued jobs through then bump up the active job limit more (2 is not enough) or for the case of atomic job you can cancel the queued atomic jobs in current kmttg session and start a second kmttg session and use FILES mode to manually run atomic & custom jobs for the relevant files. Or just be patient and let current set of jobs go through.

[chrispitude]

I have a feature request.

I'm trying to convert a bunch of kid's TV shows to .mp4 format to store on our tablet. They are cartoons so I'm able to use aggressive video compression.

In addition, I'm running the transcoding on my work laptop. It's slow and disk space is limited.

When I multi-select a bunch of these shows and start transcoding, I find that the downloads quickly outrun transcoding and the disk fills up (which ironically, then causes ffmpeg to fail too).

I would like to suggest that we provide some way to limit the number of pre-transcoding files that are not involved in any active jobs. This is purely just a disk space management issue, to minimize the number of mpeg-based files sitting around in any state (pre-decryption, post-decryption, ad cutting, qs-fix, etc.).

If transcoding outpaces downloading, then this does not apply and we can download as fast as possible. If downloading outpaces transcoding, then it would seem this disk space limiter would be always pretty handy (unless you have reasons for wanting to pull everything down ASAP).

It's not an easy problem to solve given the configurability of kmttg, but I figured I'd throw it out there to see what people thought.

[Hercules67]

Originally Posted by chrispitude

I have a feature request.

I'm trying to convert a bunch of kid's TV shows to .mp4 format to store on our tablet. They are cartoons so I'm able to use aggressive video compression.

In addition, I'm running the transcoding on my work laptop. It's slow and disk space is limited.

When I multi-select a bunch of these shows and start transcoding, I find that the downloads quickly outrun transcoding and the disk fills up (which ironically, then causes ffmpeg to fail too).

I would like to suggest that we provide some way to limit the number of pre-transcoding files that are not involved in any active jobs. This is purely just a disk space management issue, to minimize the number of mpeg-based files sitting around in any state (pre-decryption, post-decryption, ad cutting, qs-fix, etc.).

If transcoding outpaces downloading, then this does not apply and we can download as fast as possible. If downloading outpaces transcoding, then it would seem this disk space limiter would be always pretty handy (unless you have reasons for wanting to pull everything down ASAP).

It's not an easy problem to solve given the configurability of kmttg, but I figured I'd throw it out there to see what people thought.

I've run into this on one of my older PCs. Is it a big PROBLEM?

(Forgot to say, they way, I got around it, is by pre-downloading the shows to the Hard drive).

[chrispitude]

I have another question. If I check the following two boxes:

[X] Download TiVo files in Transport Stream format
[X] Combine download and tivodecode decrypt

I get an error message when I attempt to download/decrypt a file:

Super Why! - Alice in Wonderland
Recorded Wed 03/14/2012 09:00 AM on 1013=WHYYDT, Duration = 30 mins, EpisodeNumber = 156
Super Why and his friends need to get past the Queen of Hearts to catch up with the White Rabbit.
>> DOWNLOADING S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo ...
C:\kmttg_v0p8r\curl\curl.exe --retry 3 --anyauth --globoff --user tivo:MAK --insecure --cookie-jar C:\Users\Chris\AppData\Local\Temp\cookie3910098935985109635. tmp --url http://192.168.1.245:80/download/Sup...x-tivo-mpeg-ts --output "S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo"
S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo: size=2127.73 MB elapsed=0:05:54 (50.42 Mbps)
---DONE--- job=download output=S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo
>> DECRYPTING S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo ...
C:\kmttg_v0p8r\tivodecode\tivodecode.exe --mak MAK --no-verify --out "S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).mpg" "S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo"
tivodecode failed (exit code: 10 ) - check command: C:\kmttg_v0p8r\tivodecode\tivodecode.exe --mak MAK --no-verify --out "S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).mpg" "S:\Video\TiVo\Super Why - Alice in Wonderland (03_14_2012).TiVo"
TiVo Private Data : Unmatched Stream ID: No error
ts_handle_tivo_private_data failed: No error
processing frame: No error
Encryption by QUALCOMM

I did some searching and it seems these options can't be used together. Does it make sense to not allow this combination?

[moyekj]

The problem is TS .TiVo files cannot be decrypted reliably using tivodecode. So unless you have VideoRedo to use for decryption instead of tivodecode you should not enable TS downloads option.

[ThAbtO]

I have been trying to download a 15Gb file with the resume download and it always starts from the beginning.

[moyekj]

Originally Posted by ThAbtO

I have been trying to download a 15Gb file with the resume download and it always starts from the beginning.

After you set pause point on the show on your TiVo you have to refresh listings for that TiVo in kmttg or it won't have the pause location for resume.

[ellinj]

Originally Posted by moyekj

The problem is TS .TiVo files cannot be decrypted reliably using tivodecode. So unless you have VideoRedo to use for decryption instead of tivodecode you should not enable TS downloads option.

I do have VideoRedo, and have never experimented with this. What is the advantage of using TS Downloads? And how do you configure KMTTG to use VR to do the decryption?

Jeff

[ThAbtO]

Originally Posted by ellinj

I do have VideoRedo, and have never experimented with this. What is the advantage of using TS Downloads? And how do you configure KMTTG to use VR to do the decryption?

Jeff

[moyekj]

Originally Posted by ellinj

I do have VideoRedo, and have never experimented with this. What is the advantage of using TS Downloads? And how do you configure KMTTG to use VR to do the decryption?

Jeff

I recommend against using TS downloads as I had trouble with it enabled - seemed less reliable than PS downloads for me and made it so I had to reboot my Premiere to be able to download again. Some like it because downloads are faster. VRD can decrypt it as long as you have at least a partial install of TD 2.8.1 or later. I normally use VRD for decrypting myself since QS Fix & decrypt are combined into 1.

[chrispitude]

Originally Posted by moyekj

I recommend against using TS downloads as I had trouble with it enabled - seemed less reliable than PS downloads for me and made it so I had to reboot my Premiere to be able to download again. Some like it because downloads are faster. VRD can decrypt it as long as you have at least a partial install of TD 2.8.1 or later. I normally use VRD for decrypting myself since QS Fix & decrypt are combined into 1.

moyekj, can I request that you add "Requires VideoReDo" to the flyover for this TS option? It would have saved me some time. I thought having a Premiere was enough, and it was a bit of frustrating trying to make it work.

[txporter]

Originally Posted by chrispitude

moyekj, can I request that you add "Requires VideoReDo" to the flyover for this TS option? It would have saved me some time. I thought having a Premiere was enough, and it was a bit of frustrating trying to make it work.

That seems reasonable enough, although I have to ask...Why were you trying to get them in a TS container anyhow? Your posts above talk about slowing down downloads so you don't fill up your laptop before you are able to transcode the files. It would seem that moving to TS is counter to that, along with the other issues that moyekj listed above. You sound like you are playing with options that you do not understand without first trying to understand them and then are mad at the developer because he didn't protect you from yourself.

[moyekj]

Originally Posted by chrispitude

moyekj, can I request that you add "Requires VideoReDo" to the flyover for this TS option? It would have saved me some time. I thought having a Premiere was enough, and it was a bit of frustrating trying to make it work.

That wouldn't really be accurate as you can simply download TS .TiVo files without ever decrypting them in which case VRD is not needed. There are some valid reasons to leave TS .TiVo download option available (though by default it's turned off). I can probably add some more warnings related to decrypting them in the tooltip.

[moyekj]

This was already known but glad to confirm for myself with new Premiere software: The RPC remote doesn't work using the older authentication method, only works using the new PKCS12 authentication method which is in latest released version of kmttg.

[chrispitude]

Originally Posted by txporter

That seems reasonable enough, although I have to ask...Why were you trying to get them in a TS container anyhow? Your posts above talk about slowing down downloads so you don't fill up your laptop before you are able to transcode the files. It would seem that moving to TS is counter to that, along with the other issues that moyekj listed above. You sound like you are playing with options that you do not understand without first trying to understand them and then are mad at the developer because he didn't protect you from yourself.

You're exactly right.

I was just trying to go through the various kmttg options to better understand how the program works. For me, there's no point in speeding up disk transfers, yet the geek in me always wants it to be faster or better even if there's no point. I ultimately found the answer by searching the thread on the error message. I asked for a brief note to be added to the flyover to save others potential frustration.

I'm sorry I don't know as much as I should about how all this works, but I'm trying.

[OOOOPS!]

Hi First of all let me say thanks for the great program.
Now on to the problem. I've ripped some of the kids SD dvd's and would like to know if anyone has a good mp4 encoding profile for SD cartoon dvd's that I can use with pytivo to push them onto the tivo, a HDXL, in mp4? I have a whole bunch of dvd's and if left as .mpg's would fill up the tivo. ff_tivo_sd shrinks a 1.5 GB file to about 525MB but I think cartoons can be made much smaller from what I've read I just don't know enough about mp4 compression to do it.
Thanks,
OOOOPS!

[markr33]

My Get Now Playing List is always failing sometime after AnchorOffset=640. This also occurs when using TiVo Desktop. And in the process, it crashes my TiVo (Screen is blank except for TiVo background and TiVo button registers but doesn't beep) requiring a hard restart. I've tried almost everything to no avail. Help! I need some ideas of things to try and fix this.

Failure is:
... ...
>> Continuing Now Playing List from TIVO_S3_3DF5 (512/777)...
D:\Install\TIVO\TOOLS\kmttg_v0p8r\curl\curl.exe --retry 3 --anyauth --globoff --user tivo:MAK --insecure --cookie-jar C:\Users\mark33\AppData\Local\Temp\cookie348666498957510210. tmp --url https://192.168.1.95/TiVoConnect?Com...chorOffset=512 --output C:\Users\mark33\AppData\Local\Temp\NPL4329423647518654017.tm p
NPL job completed: 0:01:45
---DONE--- job=playlist tivo=TIVO_S3_3DF5
>> Continuing Now Playing List from TIVO_S3_3DF5 (640/777)...
D:\Install\TIVO\TOOLS\kmttg_v0p8r\curl\curl.exe --retry 3 --anyauth --globoff --user tivo:MAK --insecure --cookie-jar C:\Users\mark33\AppData\Local\Temp\cookie348666498957510210. tmp --url https://192.168.1.95/TiVoConnect?Com...chorOffset=640 --output C:\Users\mark33\AppData\Local\Temp\NPL4329423647518654017.tm p
Failed to retrieve Now Playing List from TIVO_S3_3DF5
Exit code: 52
Check YOUR MAK & IP settings
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 31 0 31 0 0 119 0 --:--:-- --:--:-- --:--:-- 119
0 31 0 31 0 0 119 0 --:--:-- --:--:-- --:--:-- 0
0 31 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
... ... ...
0 31 0 0 0 0 0 0 --:--:-- 0:02:00 --:--:-- 0
0 31 0 0 0 0 0 0 --:--:-- 0:02:01 --:--:-- 0
0 31 0 0 0 0 0 0 --:--:-- 0:02:02 --:--:-- 0
0 31 0 0 0 0 0 0 --:--:-- 0:02:02 --:--:-- 0
curl: (52) Empty reply from server

[moyekj]

Obviously a TiVo side problem which I have no suggestions how to fix.
If you don't care about the older shows (640+) you can limit # of shows kmttg pulls down under config->Tivos->limit # of npl fetches.

[markr33]

Thanks! I'm trying that right now.

[txporter]

Originally Posted by OOOOPS!

Hi First of all let me say thanks for the great program.
Now on to the problem. I've ripped some of the kids SD dvd's and would like to know if anyone has a good mp4 encoding profile for SD cartoon dvd's that I can use with pytivo to push them onto the tivo, a HDXL, in mp4? I have a whole bunch of dvd's and if left as .mpg's would fill up the tivo. ff_tivo_sd shrinks a 1.5 GB file to about 525MB but I think cartoons can be made much smaller from what I've read I just don't know enough about mp4 compression to do it.
Thanks,
OOOOPS!

The ff_tivo_sd profile sets the bitrate to 3000kbps. You can open up the profile (.enc files in the /kmttg/encode folder) and change the number to something smaller (or save as a new file and then tinker with it).

You could also make a new profile and try this:

# Description (single line, keep short)
<description>
mpeg4 container, same res FFMPEG CRF 19 h.264 video, copy audio (ac3)

# Encode command or script (single line)
# Known keywords: FFMPEG, HANDBRAKE, MENCODER, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
<command>
FFMPEG -y -i INPUT -threads CPU_CORES -acodec copy -vcodec libx264 -pre medium -crf 19 -g 24 -f mp4 OUTPUT

# Encoded output file extension
<extension>
mp4

That will use x264 quality based recording rather than a fixed bitrate. I use x264 (rather than ffmpeg) for my transcodes, but I also use crf 19 for SD recording. I tend to see final bitrates of 1000-1500kbps (this also includes inverse telecine and a bit of denoising), and I can't tell the difference between the DVD and H.264 quality. It should be much smaller than what you are seeing from the ff_tivo_sd profile even though it probably won't hit the 1000-1500kbps levels. I haven't actually tried the profile myself, I just made it from work here. Write back if you try and it doesn't work.

P.S. If it works for you, you can play with the CRF number to drop the bitrate even more. Normal CRF values that most folks use are in the 18-23 range.

[markr33]

Originally Posted by moyekj

Obviously a TiVo side problem which I have no suggestions how to fix.
If you don't care about the older shows (640+) you can limit # of shows kmttg pulls down under config->Tivos->limit # of npl fetches.

I tried limiting the number of shows as you described above, but kmttg is still attempting to get the list of ALL shows in NPL. And it's still failing.

If I could get the list 50 or 100 at a time, then start downloading, I would be able to eventually determine which program is the offender and DELETE it!

[ThAbtO]

Originally Posted by markr33

I tried limiting the number of shows as you described above, but kmttg is still attempting to get the list of ALL shows in NPL. And it's still failing.

If I could get the list 50 or 100 at a time, then start downloading, I would be able to eventually determine which program is the offender and DELETE it!

The number of NPL fetches is set to groups of 128 * the number set in Configuration.

0= All, 1=128, 2=256, etc.

Another thing you can try is to use java for downloads.

[moyekj]

Originally Posted by markr33

I tried limiting the number of shows as you described above, but kmttg is still attempting to get the list of ALL shows in NPL. And it's still failing.

If I could get the list 50 or 100 at a time, then start downloading, I would be able to eventually determine which program is the offender and DELETE it!

Note that setting is number of fetches, not number of shows. It used to be TiVo would fetch 128 shows per attempt but for 20.2 Premiere software that number was significantly reduced I think to around 16. So if you want to limit to 160 shows you would set it to 10. I just tried setting it to 1 and it worked fine - only performed 1 fetch which got 16 shows.
Other important thing to note: That setting is a per TiVo setting, so if you have more than 1 make sure you select the appropriate TiVo first before adjusting the number.