1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pyTivo - Transcoding server

Discussion in 'TiVo Home Media Features & TiVoToGo' started by armooo, Nov 25, 2006.

  1. Mar 8, 2007 #101 of 5684
    sabu

    sabu New Member

    17
    0
    Jan 29, 2002
    Harrisburg, PA
    With a PIII-850 - dual processor (ffmpeg appears to be running single threaded though) and a 100MB ethernet connection, I get a 1 hour show transcoded from Xvid and transferred to the TiVo (unhacked SA Dual Tuner) in about 2.5 hours. I didn't change the bitrate or any of the other settings from the initial installed values.

    I had been thinking about putting it on my file server, but I think the PII-333 in that would have to run overnight for a transfer. I may try it just for fun over the weekend.
     
  2. Mar 9, 2007 #102 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    Tested simple mods to transfer .TiVo files back. They seem to be OK:

    Two mods, both in transcode.py:

    1. Added two lines at top of tivo_compatable ():

    Code:
    def tivo_compatable(inFile):
        if (inFile[-5:]).lower() == '.tivo':
            return True
    2. Added two lines at top of video_info ():

    Code:
    def video_info(inFile):
        if (inFile[-5:]).lower() == '.tivo':
            return True, True, True, True, True 
    Before I added the 2nd mod, there were times when it would scan a folder with a .tivo in it and ffmpeg would take off running and never stop.

    This approach doesn't feed TiVo a duration (although it does send the file size) so all it has to display is the duration of the original program from the meta-data, I presume. Thus if the file is a small clip it will not show the true duration in the info display until after it has transfered completely and you look back in the NPL. I don't see this as much of a problem. That's the only difference I could detect compared to transfering with the TiVo beacon.

    See what you think --- hope I haven't built in a bug with this.
     
  3. Mar 9, 2007 #103 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    VERSION 171
    No issues, you did a great job!

    New Feature
    - Thanks to dlfl pyTivo will now transfer TiVo encrypted files as well.

    Known Issues:
    - While transfering the green play bar at the bottom of the screen will be solid green and the duration will just get longer and longer, instead of the normal thermometer effect. This is because ffmpeg is unable to read these files and cannot determine the duration. But funcationally everything else works great.

    dlfl
    Thanks again for the update, you actually solved a bug that hadn't come up yet. Specifically on windows ffmpeg could get stuck on ".tivo" files. This may also suggest the need to use a mask of known ffmpeg recognized file types. It is possible that there are other file types which could equally cause ffmpeg to hiccup.

    Where are versions 169 and 170?
    Our ever gracious benefactor, Armooo, went through and fixed my tab mistake for me :D, thank you! And made an alteration to the music plugin. He is like underwear gnome of coding, secretly fixing bugs in the night.

    Step 1: Steal Underwear
    Step 2: - - - - - - -
    Step 3: Profit!!
     
  4. Mar 9, 2007 #104 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    What frequency are you tuned to? We must be on the same wavelength ... I had exactly the same thought overnight!
     
  5. Mar 10, 2007 #105 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    I've tested a code mod to transcode.py based on the following idea:

    In video_info(), launch the ffmpeg cmd in the shell (i.e., non-blocking mode). Then wait for a maximum time (e.g., 4 secs) while polling the ffmpeg subprocess to see if it is done. If it isn't done after the max time, kill the subprocess and return None's. If it finishes before the 4 secs, stop waiting and proceed as before.

    Here's the modified code:
    Code:
        cmd = [FFMPEG, '-i', inFile ] 
        ffmpeg = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
        for i in range(80):
            time.sleep(.05)
            if not ffmpeg.poll() == None:
                break
    #    print 'ffmpeg time = ', (i*.05), ' for ', inFile
        if ffmpeg.poll() == None:
            if mswindows:
                win32kill(ffmpeg.pid)
            else:
                import os, signal
                os.kill(ffmpeg.pid, signal.SIGKILL)
    #        print 'killed ffmpeg'
            return None, None, None, None, None
        output = ffmpeg.stderr.read()
    #    print output
    Also, an "import time" statement was added near the top of the file.

    Note the commented out debug print statements I used while testing.

    I learned several things from testing:
    1. The times reported for ffmpeg to run averaged about .05 sec.

    2. I had one .tivo file that I knew locked up ffmpeg with the old code. So I commented out the code recently added to declare .tivo's OK just based on the file extension. Amazingly, with the new code it didn't lock up ffmpeg -- the kill clause never executed, i.e., just running it in a non-blocking process apparently made that difference. FYI, The output text returned from ffmpeg for this .tivo and another one was:
    :confused:

    3, Since no file was hanging ffmpeg, I had to make sure the polling and kill code actually worked. So I commented out the for loop with the sleep delays in it. Then the ffmpeg process was indeed killled for every file before it could complete, and nothing went onto the pyTivo NPL. I also verified that the info returned for non-Tivo files was still valid, i.e., identical to what it had been with the old code for these same files.

    Thus, I think this code is bullet-proof in the sense that the call to ffmpeg to get info cannot hang the program or leave ffmpeg running forever regardless of what "video" file is passed to ffmpeg.

    Obviously this needs to be tested on Linux since the changes involve os functions.

    Filtering based on file extensions may still make sense and definitely should be done if this mod is not adopted.
     
  6. Mar 11, 2007 #106 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    I had a mpeg file that I knew should be TiVo-compatible which started re-encoding when transfered. It had been edited with VideoReDo. The ffmpeg version I use (build 7215, same as Tivo.Net) reported the fps as 59.94. However there is a line in the ffmpeg return info something like:
    "Appears to be film source: 29.97..."
    The proposed patch allows the fps to be corrected back to 29.97 if this line is found.

    The patch is a few lines in the video_info() function in transcode.py. I have repeated the existing fps reading code just for context:
    Code:
        rezre = re.compile(r'.*Video: .+, (.+) fps.*')
        x = rezre.search(output)
        if x:
            fps = x.group(1)
        else:
            return None, None, None, None, None
    
        # fps can come back 59.94 on VRD-edited tivo's with ffmpeg build 7215 (TDN version)
        #  but in this case there is a line that includes "...film source: 29.97..."
        if not fps == '29.97':
            rezre = re.compile(r'.*film source: 29.97.*')
            x = rezre.search(output.lower() )
            if x:
                fps = '29.97'
    The version of ffmpeg currently being distributed with pyTivo does not have this problem. However the patch does no harm in that case and it helps those of us who might be using the later build (and I continue to hope the later build will become the recommended one).

    Oh and BTW, the file in question played fine when transfered without re-encoding.
     
  7. Mar 11, 2007 #107 of 5684
    ocntscha

    ocntscha New Member

    149
    0
    Oct 22, 2003
    I've got pyTivo going on my Linux box now. As I mentioned in a previous post the beacon was being sent out from my WAN ip address so my Tivo would never see it. I enventually came up with a solution, I changed this line in beacon.py

    self.UDPSock.sendto(self.format_beacon(), ('255.255.255.255', 2190))

    to this..

    self.UDPSock.sendto(self.format_beacon(), ('192.168.1.255', 2190))

    theorizing if I gave it the broadcast ip address of my LAN it would have to broadcast from the LAN ip address. It worked!

    The web server on port 9032 is listening on both my ip addresses and the loop back interface, I'd like to just limit it to the LAN ip but haven't figured out how. Oh well, its listenting on port 9032 of my public ip but thanks to Shorewall/iptables it won't respond to anything coming in off the internet.

    Anyhoo, seems pretty cool. Among other things, the biggest advantage it has over Tivo.Net is that new videos are just automatically available. Very nice.

    I did have a couple issues with it though, one was a video I got off youtube, a .flv file, ffmpeg would refuse to play it, kept saying bit rate of 192 is not supported for mp2 audio. I knew I could play that in Tivo.net so I "borrowed" Pipakin's settings and got it working. Actually, the one that mattered was -ar 48000, just adding that one parameter to the ffmpeg command line lets the video transcode fine. Had another one, I think the original source was PAL and it was ending up looking horizontally squished so I started playing around with the other stuff like aspect ratio.

    So, at the moment I've ended up with pretty much, if not exactly, what Tivo.Net uses.. I changed the ffmpeg line of transcode.py to this, haven't got to test a whole lot but I suspect it will do nicely, particularly since I have done a lot with Tivo.Net and have had fantastic success with it, so here's what I've got right now..

    cmd = [FFMPEG, '-i', inFile, '-vcodec', 'mpeg2video', '-r', '29.97', '-b', '4096K', '-aspect', '4:3', '-s', '720x480', '-acodec', 'mp2', '-ar', '48000', '-ac', '2', '-ab', '192', '-f', 'vob', '-' ]
     
  8. Mar 11, 2007 #108 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    Regarding needing "-ar 48000" in ffmpeg command line:

    Yep, I see the same behavior, same solution in the Windows world. Had a few .flv from google/youtube but hadn't tried them with pyTivo until now.

    Can't see how this could cause problems for other files (tested only on .wmv so far). The TiVo go back specs don't say anything about audio sampling rate and 48000 is the spec for NTSC DVD.

    Perhaps most relevant: If you load a TiVo recording into VideoReDo and look at the file parameters, the audio sampling rate is .... 48000 Hz.
    The ffmpeg doc says default sampling rate is 44000 without a -ar option. I don't understand WHY this is a problem, but.... there it is.
     
  9. Mar 11, 2007 #109 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    AND it transfers .TiVo files and TiVo-compatible MPEG2 files without re-encoding....AND one click (shortcut) gets it going ... AND ... it really does handle the end-of-transfer/file-length issue (in a way that has been fool proof in all my testing so far).

    Kind of grows on you, doesn't it? :D
     
  10. Mar 11, 2007 #110 of 5684
    sabu

    sabu New Member

    17
    0
    Jan 29, 2002
    Harrisburg, PA
    Well, the PII-333 over 100Mbit Ethernet only took 6 hours to transfer a 1 hour 15 minute show. Not as bad as I thought it would be.
     
  11. Mar 12, 2007 #111 of 5684
    armooo

    armooo pyTivo Developer

    81
    0
    Feb 1, 2003
    I added a bunch of the above patches and fixed one bug that should be a large speed up for anyone who's tivo dose not have a domain name. (4.5 sec per request speed up)

    pyTivo
    - Don't do reverse DNS lookups when logging (adds 4.5 sec to response time)
    - Check for Appears to be film source: (\d+) and override fps with it
    - Set an audio sample rate
    - Caching video info
    - Timeout for ffmpeg getting video info
     
  12. Mar 12, 2007 #112 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    Thanks armooo ! The speedup is very welcome!

    Glad to see the upgrade to build 8047 of ffmpeg too.

    I think I've learned a python lesson (remember I'm a newbie to this language):
    Apparently the blocking after the video_info() call of the ffmpeg subprocess comes from the ffmpeg.stderr.read () statement, not from the Popen call.

    One slight problem:
    "- Check for Appears to be film source: (\d+) and override fps with it" doesn't work with build 8047 -- it returns a different string than 7215, so the "...film source: 29.97" doesn't match anything.

    I wonder if trying to chase this around for different builds of ffmpeg is worth it? At least for the file I've been testing with (which is 352x480) the re-encoded version has the correct aspect ratio and looks pretty good.

    Also, at least on my python (windows, 2.5) the RE "(\d+)" doesn't match "29.97" because the \d doesn't match the decimal point (or period). I would have to use "(\d+)\.(\d+)" and combine the two groups. Am I missing something?

    On this patch (if you decide to keep it), I still prefer gating the whole patch with
    if not fps == '29.97':
    because the sole purpose of the patch is to allow a chance to correct the fps to 29.97 if the usual matching returned something else. (Actually I prefer my origiinal patch, which required an explicit 29.97 match for the override.)

    If you decide to keep the patch, I will be happy to supply the additional patch to make it work with build 8047 if you would like. The patches are accumulative, i.e., you don't have to remove the one for 7215 when you add the one for 8047.
     
  13. Mar 12, 2007 #113 of 5684
    ocntscha

    ocntscha New Member

    149
    0
    Oct 22, 2003
    You guys rock! Getting every last little gremlin out, I love it.

    I haven't actually tried this latest release but I did download and look at it. I see you've now got -ar 44100 in the ffmpeg command line. According to the ffmpeg man page 44100 is the default for that parameter anyway. Changing it to 48000 is what enables youtube .flv files to work for both dlfl and I, he details it pretty thoroughly a few posts back.

    Thank you for creating pyTivo and sharing it with us.
     
  14. Mar 12, 2007 #114 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    I noticed this too, but it transcoded one of my .flv files OK with 44.1k. I wonder if the ffmpeg doc is just not correct on this -- or it's something that varies with the many builds. Still -- seems like 48k generally would be prefered since that's what TiVo files have. :confused:

    This thing is really getting sweet, isn't it!
     
  15. Mar 12, 2007 #115 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Sorry I have been neglecting you all. And I am glad to see Armooo is more then happy to pickup the slack.

    I only have one concern with the above modifications. And it is a minor one. Setting the timeout on the ffmpeg loop to .05 seconds means that even if a users computer operates faster than .05 seconds he will still be limited by this loop.

    Like I said, it is only a minor concern, and probably doesn't affect anyone in any form that is noticable.

    Other than that I am very excited about working out all the little bugs.

    The one main bug I would still like to fix:
    - Moving the LRU cache from video.py to transcode.py into the video_info() block.
    OOPS: WAHOO Armooo already did this. I guess I should pay attention more. Thanks Armoo

    Future Features:
    - After moving the LRU cache to video_info(), add the option to scan all subdirectories on start up for video files.
    - Some how add titles and descriptions and maybe other data to the videos.

    What is this new dll??
    Armooo, what is this new dll in the plugins/video folder?? pthreadcg2.dll??
     
  16. Mar 12, 2007 #116 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    From having tried ffmpeg build 8047 previously, I know this dll must accompany it for it to operate. (You find out by trying to run it without the dll in the path and it tells you it can't find that dll!). IIRC, the dll is downloadable from the same site as the ffmpeg build but just up one directory.

    Also: I understand your concern about the .05 sec ffmpeg wait.... who knows? It's a judgement call. The computer I tested on is no slouch (3 GHz P4, hyper-threaded) and it was averaging about .05 secs (some zeros, some .1 and an occasional .15). It was probably the cache changes Armooo made that made the difference, but my NPL list is populating MUCH faster than before -- very satisfying!
     
  17. Mar 12, 2007 #117 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    I'm going to post my proposed changes in transcode.py (starting from ver. 173) to handle the case where a tivo-compatible file has fps reported as 59.94 by ffmpeg, either Build 7215 (Tivo.Net version) or Build 8047 (current pyTivo version).

    In video_info() replace:
    Code:
        rezre = re.compile(r'.*film source: (\d+).*')
        x = rezre.search(output.lower())
        if x:
            fps = x.group(1)
    with:
    Code:
        # fps can come back 59.94 on VRD-edited tivo's with ffmpeg build 7215 (TDN version)
        #  or with build 8047 (latest pyTivo distribution) but in this case there 
        #  is a line that includes "...film source: 29.97..." for build 7215 or 
        #  one that contains "...frame rate differs from container frame rate: 29.97..."
        #  for build 8047.
    
        # Allow override only if it is mpeg2 and frame rate was doubled to 59.94
        if (not fps == '29.97') and (codec == 'mpeg2video'):
            # First look for the build 7215 version
            rezre = re.compile(r'.*film source: 29.97.*')
            x = rezre.search(output.lower() )
            if x:
                fps = '29.97'
            else:
                # for build 8047:
                rezre = re.compile(r'.*frame rate differs from container frame rate: 29.97.*')
                x = rezre.search(output.lower() )
                if x:
                    fps = '29.97'
    I'm not trying to force these on anyone and I realize mssrs. KRKeegan and Armooo may prefer to ignore or change them. I just have this strange desire to share. :D

    Bon appetit!
     
  18. Mar 13, 2007 #118 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    I posted on the Tivo.Net thread suggesting people having problems with it give pyTivo a try. So far two people (see Posts #1147 and #1151 on that thread) have responded saying they couldn't get pyTivo installed and working. I posted back suggesting they post pyTivo problems on THIS thread since I don't want to hijack the Tivo.Net thread.

    BTW, is pyTivo known to work on MAC's ?
     
  19. Mar 13, 2007 #119 of 5684
    Chew

    Chew New Member

    284
    0
    Jan 22, 2003
    ^ To answer your question from the TiVo.net thread, those three attempts to get it installed (and it failed everytime) occured just this past weekend.

    I'll give it another go after work and post exactly where it gets hung up.
     
  20. Mar 13, 2007 #120 of 5684
    dlfl

    dlfl Cranky old novice

    6,997
    18
    Jul 6, 2006
    Near...
    Good deal. If you have problems again, please post your computer specs and a copy of the pyTivo.conf file you used. (If you didn't edit this file, that would explain pyTivo not running. The required edits are very simple however.)

    You have to install python version 2.4 or later. (Why not 2.5 ?). To verify installation select the command line version from the start menu and type in something simple like: a = 'hello' and hit the return key.

    I've never used the pyTivo windows installer. AFAIK there are no registry mods for pyTivo so using the Windows installer isn't important. There is a .zip file with the same version number. Just put everything in the zip file into a folder of your choice. Edit the pyTivo.conf file. Launch program by double clicking the pyTivo.py file (or a shortcut pointing to it). You may see nothing in the command window that pops up until you look at the NPL in tivo, or you may see a long warning about a Cheetah something or other not being installed -- ignore this warning.

    (I've been assuming you are on a Windows system.)
     

Share This Page