1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

New program for 1 step TTG downloads, decryption, encoding - kmttg

Discussion in 'TiVo Home Media Features & TiVoToGo' started by moyekj, Mar 15, 2008.

  1. Jun 19, 2010 #2421 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    Gremlin,

    Thanks. Does kmttg using Java 1.5.0_24 on your 10.5.8 system have any issues like I'm experiencing with short bursts of activity interspersed with unexplained "dropouts" in responsiveness, etc.?

    Today I restarted kmttg from scratch and told it to save metadata, download, decrypt, and extract closed captions for a 9 GB recording on a TiVo Premiere that uses a wireless-N network connection. This was at 10:37 AM. It's now 8:34 PM and the decryption job is crawling along, about 2/3 done. The download took over 5 hrs., from 10:37 AM to 3:59 PM!

    I did not have any other kmttg jobs queued during this entire time.

    I'm pulling my hair out ... all this used to work fine.

    I'm tempted to do a clean install of OS X 10.5 next and see whether my problem gets cleared up by that!
     
  2. Jun 19, 2010 #2422 of 10413
    moyekj

    moyekj Well-Known Member

    11,148
    33
    Jan 23, 2006
    Mission...
    Sounds to me like you have disk I/O problems. If you take that 9GB recording and copy it to a different file name on same disk does it also take much longer than you would expect?
     
  3. Jun 20, 2010 #2423 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    moyekj,

    Excellent guess!

    I am currently trying making a duplicate of my 9GB file to the same external drive. It's taking roughly 50-60 minutes to wrap up. I calculate its overall transfer rate at about 3MB per second -- though of course it is reading and writing data, so maybe I should double that.

    Whether or not I double it, it's way below the nominal 480MB/sec of USB 2.0, which is the interface the external drive uses.

    I need to look into what's wrong here. I've got five external hard drives (and a printer and SoundSticks) on two 4-way USB 2.0 hubs hooked to my Mac. One of the hubs is new, as is one of the drives. Three of the drives, including the newest one, are e-SATA drives sitting in BlacX docking stations. This general arrangement has been working fine for me, but maybe the new hub, or drive, or docking station has caused something to break. Further experimentation is in order, after which I'll report back ...

    Edit: OK, I found out what the problem was. Oops ... I was nearly out of space on the external drive! Any time I would try to use that drive, kmttg would have short bursts of activity interspersed with longer periods of non-responsiveness. I didn't notice how low on drive space I was, nor did I notice that kmttg has a config setting for Min Requested Space (GB) that, had I been using it, would have alerted me to the situation.

    Thanks for your excellent input!
     
  4. Jun 22, 2010 #2424 of 10413
    AudioNutz

    AudioNutz New Member

    318
    0
    Nov 9, 2008
    I'm so glad that you've got it sorted! It's summer here, so I have not been able to spend time on this. I was going to report back that I have a "Leopard Server" running Mac OS X 10.5.8, with Java 1.5.0_22, and then I've also got a "Snow Leopard Server" running Mac OS X 10.6.0, with Java 1.6.0_15.

    I was also going to have you quit everything that has to do with kmttg, and try the curl command by itself, to see if that raises your issue to the surface again. (It would have ruled out kmttg and the distributed pieces, since curl is embedded in the OS)

    The other thing I would have mentioned is that I'm still back on the "j" version of kmttg... (For the same reason above, it's summer and I haven't got much time to fool with this.)
     
  5. Jun 23, 2010 #2425 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    Thanks for your concern and input, AudioNutz. Actually, I was not aware that curl can be executed in standalone fashion! That's what comes of not looking hard enough at the messages output by kmttg. Right at this moment there's the following message showing a curl call by kmttg:

    Code:
    /usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie2423857590188571200.tmp --url http://10.0.1.7:80/download/The%20Story%20of%20India.TiVo?Container=%2FNowPlaying&id=1601172 --output "/Volumes/Hitachi 2TB Drive/Downloads/The Story of India - Beginnings (06_11_2010).TiVo" 
    I'm not sure what all those options do, but clearly --url specifies the URL of the particular TiVo file I'm downloading, complete with the IP address my TiVo is currently using, 10.0.1.7. --output gives where the output .TiVo file is to be put. --user tivo:MAK gets filled in, I assume, with my media access key.

    Anyway, I've learned my lesson: pay attention to how low on space my destination hard drive is getting!

    Cheers,
     
  6. Jun 23, 2010 #2426 of 10413
    AudioNutz

    AudioNutz New Member

    318
    0
    Nov 9, 2008
    This is really strange, because the versions of kmttg that I use have an automagic check to see if the drive space is low. Maybe this is just for "Auto" tasks...
     
  7. Jun 23, 2010 #2427 of 10413
    Phantom Gremlin

    Phantom Gremlin New Member

    1,555
    0
    Jun 20, 2002
    Tualatin,...
    The problem has been solved, so this is a suggestion for others who may stumble upon this thread when searching ...

    In addition to curl, one other technique is to use a browser to go directly to https:<tivo_ip_address> and download that way. IIRC the username should be tivo and the password should be your MAK.

    The idea is that when more complicated programs like kmttg have problems, try simpler techniques, it makes troubleshooting easier.
     
  8. Jun 24, 2010 #2428 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    I've noticed that kmttg removes the punctuation marks from the title of recordings it downloads -- apostrophes, question marks, and so on. I wonder whether there shouldn't be a config option to prevent this? It's a pain putting the punctuation marks back in, and, after all, only ':' isn't allowed in file names. Thanks for listening ...
     
  9. Jun 24, 2010 #2429 of 10413
    orangeboy

    orangeboy yes, I AM orangeboy!

    4,083
    0
    Apr 19, 2004
    East Moline, IL
    Unless you are running Windows...
     
  10. Jun 24, 2010 #2430 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    I'm on a Mac. The situation with Windows is why I suggest that leaving punctuation characters as is ought to be user-configurable. Or, alternatively, kmttg could check to see what operating system it's running on, and act accordingly ...
     
  11. Jun 24, 2010 #2431 of 10413
    AudioNutz

    AudioNutz New Member

    318
    0
    Nov 9, 2008
    Even on the Mac, kmttg makes calls to several "Command Line" tools. These are not always friendly when it comes to punctuation. If you read back in this thread far enough, I'm sure you'll find the part where this issue was discussed.
     
  12. Jun 25, 2010 #2432 of 10413
    nickhaas33

    nickhaas33 New Member

    38
    0
    Dec 11, 2006
    This may have been asked before, in case it has, I apologize.
    What is the trick in FFMPEG to force voice synchronization with the movie.
    I inevitably end up with the voice about a second out of synch by the end of encoding.
    I use the following command line that works great for the low volume problem I had on my Archos player, but the voice does get out of synch (which I thought -async 1 prevented). It is still watchable, but irritating.

    FFMPEG -y -i INPUT -threads CPU_CORES -vcodec mpeg4 -maxrate 2000k -qmin 3 -qmax 5 -bufsize 5096k -g 300 -acodec libmp3lame -ac 2 -ab 128k -ar 48000 -vol 1280 -s 720x480 -r 29.97 -b 1500k -async 1 -f mp4 OUTPUT

    Thanks
    Update:
    I saw this on a Linux site, so maybe I do need another value:
    -async 1 is a special case where only the start of the audio stream is corrected without any later correction.

    Does anyone know a fool-proof number for async?
     
  13. Jun 25, 2010 #2433 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    OK, I'll look for those past posts. I did do a search in the thread on "punctuation" but got no results.

    But I also have to note that there is really no problem at all on the Mac command line with file names containing punctuation. Just enclose the whole, fully qualified file name in double-quote characters. Then, the only restrictions become those that the Mac OS itself imposes, e.g., '/' characters are not allowed in file names. Neither are ':' characters in the Finder — as the two mean the same thing, with '/' in the command line and ':' in Finder both representing the separator between directory levels or between the directory and the filename proper.

    I have to believe that kmttg already puts double quotes around filenames, since they're the easiest way to allow spaces to appear in a filename. So I imagine kmttg is simply filtering out such characters as apostrophe and question mark to cater to OS's that don't permit them. So why not make that filtering process configurable? On a Mac, the user could optionally choose to bypass it. Users on other OS's could leave it alone. No harm, no foul ...
     
  14. Jun 25, 2010 #2434 of 10413
    orangeboy

    orangeboy yes, I AM orangeboy!

    4,083
    0
    Apr 19, 2004
    East Moline, IL
    So on a Mac, you can have a path that looks like this?:
    C\Path\To\Some\Folder\File.extension

    and/or this?:
    C:path:To:Some:Folder:File.extension

    I'm sure my Mac ignorance is blatantly showing... :eek:
     
  15. Jun 25, 2010 #2435 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    /Path/To/Some/Folder/File.extension

    is valid on the Mac. The separators are forward slashes, not backward slashes. The first '/', starting from the left, stands for the root directory, as understood by Unix.

    There are no drive letters, either. You specify which hard-drive volume to use with something like:

    "/Volumes/New 2TB Drive/TiVo Transfers/Just Transferred/Mr. Hollands Opus (06_23_2010).TiVo"

    Instead of a drive letter, the whole thing starts with /Volumes/New 2TB Drive/... .

    (Notice that the name of the movie has been changed by kmttg to "Mr. Hollands Opus," with no apostrophe. That's what I'm basically objecting to.)

    The syntax containing ':' characters is not actually used on the command line, but it is not allowed by the Finder, which is sort of the Mac's equivalent to the whole gamut of Windows windows that show the contents of directories.

    Get it? The Finder is part of the Mac OS X operating software, but it is not itself Unix. Mac OS X (and Finder) sit atop Unix, which is what you are dealing with in the command line itself.

    So you never see any syntax with ':' characters, anywhere. But Finder internally turns Unix '/' characters into ':', so it won't let you embed ':' in visible filenames.

    It sounds complex, but as long as kmttg were to put the filenames all in double-quotes and avoid internal double quotes, '/' characters, and ':' characters, fully qualified Mac filenames would come out fine.
     
  16. Jun 25, 2010 #2436 of 10413
    AudioNutz

    AudioNutz New Member

    318
    0
    Nov 9, 2008
    I'll let Kevin chime in here. I think the apostrophe's were getting replaced with some other character automagically during the download process, because of some issue with curl. The remaining commands were attempting to execute with the correct name with the apostrophe, but since the name was truncated by curl... It couldn't find the file to perform the work on.
     
  17. Jun 25, 2010 #2437 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    OK, thanks, I've found the post that talks about it, #487 in this thread. (Is Kevin moyekj, BTW?)

    So, is curl really the critter responsible for dropping the apostrophes? (And, I assume, other characters such as question marks ... ) I ask because I recently downloaded a program called

    Closer to Truth - Why Is Consciousness Baffling?

    that has a question mark. I saw in kmttg's messages:

    Code:
    Closer to Truth - Why Is Consciousness Baffling?
    Recorded Mon 06/21/2010 08:30 AM on 268=WMPTDT2, Duration = 30 mins
    Understanding inner awareness. 
    OVERWRITING EXISTING FILE: /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt
    
     ...
    
    >> CREATING /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt ...
    /usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie2199505577978024301.tmp --url https://10.0.1.7:443/TiVoVideoDetails?id=1651715 --output /tmp/meta8375990166995294744.tmp 
    metadata job completed: 00:00:01
    ---DONE--- job=metadata output=/Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).mpg.txt
    
     ...
    
    >> DOWNLOADING /Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).TiVo ...
    /usr/bin/curl --anyauth --globoff --user tivo:MAK --insecure --cookie-jar /tmp/cookie8252577655988769831.tmp --url http://10.0.1.7:80/download/Closer%20to%20Truth.TiVo?Container=%2FNowPlaying&id=1651715 --output "/Volumes/Hitachi 2TB Drive/Downloads/Closer to Truth - Why Is Consciousness Baffling (06_21_2010).TiVo" 
    
    That OVERWRITING EXISTING FILE message suggests to me that the '?' was dropped before curl was called!

    If the same logic applies to apostrophes, then are we sure the apostrophes disappear only at the time curl is called?

    Of course, it may be the case that the punctuation characters have to be made by kmttg to vanish at (or prior to) the curl call simply to get curl to work ... but, if so, is there any reason they can't (optionally, at user discretion) be stuck back into the filename by kmttg after curl has done its duty and the downloaded TiVo file is sitting there on the hard drive? I.e., can't kmttg optionally rename the TiVo file, once it's been downloaded, and use the revised name that contains all the original punctuation characters in subsequent jobs such as decrypt.

    I note also that the apostrophe in "Mr. Holland's Opus" and the question mark in "Why Is Consciousness Baffling?" are preserved in the metadata files created by kmttg.

    Of course, a lot of what I'm saying is guesswork on my part, and I don't really have any idea how easy or hard it would be to give Mac users of kmttg a "preserve punctuation" option implemented via file-renaming logic. In view of the fact that kmttg is multi-platform software, my idea might adversely impact users of other platforms. It might just be too hard to implement my idea without going to a lot of trouble to re-code affected areas within kmttg, so as to enable this "preserve punctuation" option that would cater to only a small subset of users anyway.

    Anyway, that's my two cents ...
     
  18. Jul 2, 2010 #2438 of 10413
    moyekj

    moyekj Well-Known Member

    11,148
    33
    Jan 23, 2006
    Mission...
    epstewart, special characters in file names for Unix/Mac systems is not as trivial as you may think to support, especially in the context of making system calls via Java. kmttg basically uses Runtime.getRuntime().exec(String []) to execute system calls and characters such as single or double quotes, slashes, backslashes and a few others in that context do cause issues. Probably with quite a bit more work and some OS-specific considerations the allowed character list per OS could be expanded, but it's really a can of worms I don't feel like opening. The list of characters that are filtered out now were mostly born out of past issue reports.

    If you feel strongly about it then source code is available to make whatever changes you may want to try. The place to start looking would be removeSpecialChars method in main/tivoFileName.java. This is what filters out characters in file names before any task in kmttg is called (so yes the characters are stripped out before calling curl, etc).
     
  19. Jul 2, 2010 #2439 of 10413
    epstewart

    epstewart Member

    648
    0
    Mar 1, 2003
    Thanks for a very helpful reply. I gather, then, that the difficulty in making system calls via Java using strings with pesky special characters in them is the root cause for the need to suppress those characters in output filename literals.

    I have no knowledge of writing Java code, so I am totally naive here. Still, I am wondering if there is a way to "escape" special characters such as ? and ' in the literals in question. As you may know, in Unix the backslash (\) is generally used to designate that the following character is to be taken literally, as part of the string itself. So \" allows a double quote to be included in a string that is delimited by " characters, for example. Likewise, backslash can be included via \\. Using such escape sequences, any character can be included in any literal string.

    I would think there must be a way to do the same kind of thing for strings used in Java system calls, no?
     
  20. Jul 2, 2010 #2440 of 10413
    Icarus

    Icarus New Member

    360
    0
    Jun 15, 2002
    Upcountry...
    It's not unix, it's the shell that does that.

    When I run bash or tcsh (or whatever your favorite shell is) on my windows system (under cygwin, for example) I can do the same thing when I type those characters, but it's the shell that processes them. They don't get passed into library calls or system calls.

    -David
     

Share This Page