1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pyTivo - Transcoding server

Discussion in 'TiVo Home Media Features & TiVoToGo' started by armooo, Nov 25, 2006.

  1. Mar 4, 2007 #81 of 5684
    Zothar

    Zothar New Member

    2
    0
    Mar 4, 2007
    Here are the lines where I see it:
    Code:
    46:    type, width, height, fps, millisecs =  video_info(inFile)
    110:    type, width, height, fps, millisecs =  video_info(inFile)
    144:    if video_info(inFile)[0]:
    145:        return video_info(inFile)[4]
    
    For video, the only use of LRU that I see currently is in video.py to keep the playable or not boolean. video_info() itself should probably do an LRU internally, which would keep the other code simple.
     
  2. Mar 4, 2007 #82 of 5684
    Jabo4

    Jabo4 New Member

    21
    0
    Feb 17, 2007
    Yea I'm not sure what's up with the service. It just spits out the usage stuff when I run it. I can try start/stop install/update/remove w/e it just does nothing. Not a big deal I may write a service myself in .net to accomplish it. Thanks for all the hard work!
     
  3. Mar 4, 2007 #83 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Hmm, I am trying to reproduce this. I have a TiVo file that is 352x480 and it worked properly.

    Can you tell me the following details about the video:
    Type: I am assuming mpeg2video, but just confirm for me
    FPS
    Dimensions

    And I will try and figure out what is going on.

    Kevin
     
  4. Mar 4, 2007 #84 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Here is what mediaInfo says:

    This was edited out of a Best-Quality TiVo recording using VideoReDo. I'm pretty sure these specs are exactly those of the original TiVo file. VideoReDo does no re-encoding -- ever.

    If you load a file in VideoReDo it will tell you specs too:
    For the original TiVo recording you get exactly the same thing except the name and duration are different (expected) but also the Display Size is 720x480. However the mpeg2 plays with correct AR in WMP11 and when transfered to the TiVo via TiVo go back.

    Actually, I remember having this same aspect ratio problem initially with TiVo.net (which always re-encodes even TiVo compliant mpeg2's as you know). It was in response to this that Pipakin added the two choices for aspect ratio. I think if you can keep it from re-encoding the problem might vanish. I'm wondering if the "Display Dimensions" are being reported by ffmpeg to pyTivo and since they are not 720x480 pyTivo thinks it needs to be re-encoded? This could be a difference between versions of ffmpeg -- I'm using the one that comes with TiVo.net.
     
  5. Mar 4, 2007 #85 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    I've tested with a short DivX file and a 1.5 hour AVI (XVID + mp3). No end-of-file problems.

    Score one for Zothar's aspect ratio patch, however. The AVI file was a 320x192 and without the patch it would not encode, throwing a top crop error (IIRC). With the patch it encoded fine. It ended up stretched about 9% vertically, but this probably would not be noticed by most viewers. I determined this by measuring a stop sign in one scene on the TV screen. When transfered with TiVo.net (with the correct aspect ratio choice) it was compressed vertically by about 4% -- I'm not sure my TV is any better than that.
     
  6. Mar 4, 2007 #86 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Ok sorry folks I am dead tired tonight. I was up at 4:30 for a bike tour before the marathon here in LA.

    Zothar Alright I will agree to the bad algebra in the vertical padding. But I double checked the horizontal and I think I have that one correct. I don't know look at them both again and let me know. It has been a long time since I had algebra.

    Update
    Other than fixing that minor bug. I didn't do anything else tonight. dlfl I will do my best to figure out why ffmpeg is still transcoding a TiVo file. Zothar I tried to move LRUcache from video.py to transcode.video_info(), but it just gave me a headache and I think I am too tired to think properly. Jabo4 Once I move the cache to it's new location I think I can create a quick extra file that can optionally be spawned at start to query all files.

    So maintenance update tonight, nothing big. Have a good night all.
     
  7. Mar 5, 2007 #87 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    @KRKeegan
    Sleep well, oh good and faithful servant! :cool:
     
  8. Mar 5, 2007 #88 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Since I am using a different ffmpeg, it could be as simple as using Zothar's ffmpeg patch, or a similar problem related to the way ffmpeg reports file specs. I haven't looked at the patch yet.
     
  9. Mar 5, 2007 #89 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    The patch is in 167, but i dont think that is your problem. pyTivo determines by a list if a file is compatible. The only thing I can think of is that the dimensions are not in the list.
     
  10. Mar 5, 2007 #90 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    The 480x480 is already in the table. However the format of the line returned from ffmpeg is different for the one I'm using and it messes up the return from video_info().

    pyTivo-supplied ffmpeg:
    ffmpeg I'm using:
    I changed transcode.py as follows:
    Code:
        rezre = re.compile(r'.*Video: (.+), (.+), (\d+)x(\d+), (.+), (.+) fps.*')
        m = rezre.search(output)
        if m:
            millisecs = ((int(d.group(1))*3600) + (int(d.group(2))*60) + int(d.group(3)))*1000 + (int(d.group(4))*100)
            return m.group(1), int(m.group(3)), int(m.group(4)), m.group(6), millisecs
    and it returned the correct info. Then a TiVo-compatible mpeg transfered without re-encoding and played with the correct aspect ratio.

    However there was still a problem. The clip was 3 min 52 sec long and it stopped transfering at about 80% and the following error message appeared in the pyTivo command window:

    Note the first line was not part of the error message -- I just added a print statement in tivo_compatable() to indicate what info was being parsed from ffmpeg. Note the millisecs value is correct for the clip duration.

    I repeated the transfer just to verify it did exactly the same thing. The part that transfers before the error does stay in the NPL and can be played.

    I can't help suspecting this error is a result of underestimating the file size (???).

    EDIT:
    It was the estimate, I think.
    I modified video.py as follows:
    Code:
            def est_size(file):
    	    #Size is estimated by taking audio and video bit rate adding 2%
    	    return int((duration(file)/1000)*((6200 * 1.02 * 1000)/8))
    (i.e., substituting 6200 for the 4288 that was there).
    The entire file transfered -- after a short while the 206 response was sent, and all was well.

    This was done because this clip was recorded at TiVo Best Quality which is around 5800 kbps. Of course this is just a test hack and probably shouldn't be left that way -- it overestimates re-encoded files by way too much. There needs to be a more accurate file size estimate for TiVo-compatible files, right ?
     
  11. Mar 6, 2007 #91 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    I've modified video.py and transcode.py (from Version 167) with two goals in mind:

    1. Adjust estimated file size properly for TiVo-compatible mpeg2 files that will be transfered without re-encoding. The actual bitrate reported by ffmpeg is used in the estimated size computation for those files. (For re-encoded files the already existing computation continues to be used.) Without this fix, a mpeg with bitrate of around 6000 kbps (e.g., TiVo Best Quality) would not transfer completely, and lower bitrate files (e.g., TiVo Medium Quality) would have greatly overestimated file sizes.

    2. Make the video_info() function, which parses the data reported by ffmpeg, more robust to variations of format due to either ffmpeg version used or type of file. This has been tested on the pyTivo-distributed ffmpeg and on the TiVo.Net ffmpeg and on several different files.

    Code mods are as follows:

    In transcode.py, the tivo_compatable() and video_info() functions were modified as follows:

    Code:
    def tivo_compatable(inFile):
        suportedModes = [[720, 480], [704, 480], [544, 480], [480, 480], [352, 480]]
        type, width, height, fps, millisecs, bitrate =  video_info(inFile)
        #print type, width, height, fps, millisecs, bitrate
    
        if not type == 'mpeg2video':
            #print 'Not Tivo Codec'
            return 0
    
        if not fps == '29.97':
            #print 'Not Tivo fps'
            return 0
    
        for mode in suportedModes:
            if (mode[0], mode[1]) == (width, height):
                #print 'Is TiVo!'
                return bitrate
            #print 'Not Tivo dimensions'
        return 0
    
    def video_info(inFile):
        cmd = [FFMPEG, '-i', inFile ] 
        ffmpeg = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
        output = ffmpeg.stderr.read()
        durre = re.compile(r'.*Duration: (.{2}):(.{2}):(.{2})\.(.), (.+),.*bitrate: (\d+)')
        d = durre.search(output)
    
        rezre = re.compile(r'.*Video: ([^,]+),.*')
        x = rezre.search(output)
        if x:
            codec = x.group(1)
        else:
            return None, None, None, None, None, None
    
        rezre = re.compile(r'.*Video: .+, (\d+)x(\d+),.*')
        x = rezre.search(output)
        if x:
            width = int(x.group(1))
            height = int(x.group(2))
        else:
            return None, None, None, None, None, None
    
        rezre = re.compile(r'.*Video: .+, (.+) fps.*')
        x = rezre.search(output)
        if x:
            fps = x.group(1)
        else:
            return None, None, None, None, None, None
    
        millisecs = ((int(d.group(1))*3600) + (int(d.group(2))*60) + int(d.group(3)))*1000 + (int(d.group(4))*100)
        return codec, width, height, fps, millisecs, int(d.group(6))
    The video_info() function uses separate regular-expression searches for video parameters where a single search was used before. It also returns an additional (6th) parameter, which is the actual bitrate.

    The tivo_compatable() function no longer returns True or False. It returns 0 if the file is not TiVo compatible but returns the actual bitrate otherwise.

    The select_aspect() function also needed a trivial modification because of the extra parameter returned by video_info ().


    In video.py, the est_size() function is modified as follows:
    Code:
            def est_size(file):
                full_path = os.path.join(path, file)
                bitrate = transcode.tivo_compatable(full_path)
               #Size is estimated by taking audio and video bit rate adding 2%
                if bitrate > 0:  # Is TiVo compatible mpeg2
                  return int((duration(file)/1000)*((bitrate * 1.02 * 1000)/8))
                else:  # Must be re-encoded
    	      return int((duration(file)/1000)*((4288 * 1.02 * 1000)/8))
    Thus if the file is TiVo compatible, the bitrate returned by tivo_compatable will be greater than zero and it will be used instead of the constant 4288 in the computation.

    This tests fine in my limited testing so far but I am definitely not an experienced Python programmer so ....... :rolleyes:
     
  12. Mar 7, 2007 #92 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Verified my code mods (previous post) work with two additional versions of ffmpeg: 7760 and 8047.
     
  13. Mar 7, 2007 #93 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Sorry I have been busy the past few days.

    Thank you for the help though. All of you modifications seem good, except the file size of TiVo Compatible files. There is nothing wrong with it, in fact the way you handled it is very ingenius. However, since we are just sending a whole file the best way to handle this would just be to get the exact file size and use that. This means TiVo doesnt have to get into that whole 206 issue. Not a big deal, but probably the more proper thing to do.

    I will add your changes tonight and hopefully change the TiVo compatible file size too.

    Thanks for you patience everyone. And you exhaustive testing :p
     
  14. Mar 8, 2007 #94 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    ok again another minor update to fix some small bugs.

    New in this version:
    - More versions of ffmpeg output supported
    - TiVo compatible files are reported to TiVo as actual size.

    Thanks to dlfl for the bug testing and solutions.

    I made a change to your file size solution though. As previously discussed I report TiVo compatible file sizes as there actual file size. This circumvents any need to get into that whole 206 area. And it is the proper way to transfer a non-transcoded file. Who knows it may prevent issues in the future.

    Future planned minor update(maybe tomorrow)
    Transfer of .tivo files. Since ffmpeg can't be used to verify correct format we will just have to assume that all ".tivo" files are compatible. A reasonable assumption.

    Alright night all.

    Kevin
     
  15. Mar 8, 2007 #95 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Yeah, I thought of that first but being the newbie to Python that I am, I couldn't figure out how to get the file size ! :eek: That is obviously the best way.

    All I could think of was an fseek, ftell sequence. I'm guessing there is a better way to get the file size.

    I don't know what's going on with Pipakin..... I've kind of gotten to like pyTivo anyway. All I do is start it with a shortcut and it's off and running -- actually simpler than TiVo.Net and serves my purposes fine.

    I do prefer the ffmpeg that Tivo.Net uses ... I think it's significantly better than the current one. It handles wmv3 video (encoded with WMV9 codec). It is build 7215 and the Windows version is downloadable from this web page -- it is the one named ffmpeg-SVN-r7215-static-gpl-win32.zip Of course it is just called ffmpeg.exe in the zip. If you run "ffmpeg -i dud" you get all the build information and it is identical for this one and the TiVo.Net one, including the build date/time right down to the second. I just tested by substituting and it worked fine although I only tested it on one WMV3 video. I've tested the Tivo.Net one on a wide variety of formats.

    Glad to see you back "on the job" again. :)
     
  16. Mar 8, 2007 #96 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Just tested 168 on one TiVo compatible and one WMV. Worked fine! I did notice some indenting irregularities in est_size () and corrected them before running. In this case the logic would work the same whether the last return statement was indented or not I think, i.e., whether it was in the else: clause. The first return was only indented 3 spaces (all other indents are 4) and I don't know enough about Python to know how important that might be so just made both indents 4 spaces.

    Your plans to handle .tivo sound good. :up:
     
  17. Mar 8, 2007 #97 of 5684
    sabu

    sabu New Member

    17
    0
    Jan 29, 2002
    Harrisburg, PA
    Is there a minimum machine spec required for running pyTivo?

    Will an older, slower machine still work but just have the transfer take longer because the transcoding takes longer or could it be too slow and timeout the process?
     
  18. Mar 8, 2007 #98 of 5684
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Exactly, as long as you can run python2.4 and ffmpeg, which is I think just about any machine, you can run pyTivo.

    You will however not be able to watch your videos while they stream live however.

    It might be nice for people to post their Network, CPU speeds and the rate at which pyTivo is able to transfer an average file??

    For example I have a 2.8Ghz processor, running 100Mbps wired ethernet and I can stream an average downloaded TV show at about 125% of real time.
     
  19. Mar 8, 2007 #99 of 5684
    dlfl

    dlfl Cranky old novice

    6,998
    18
    Jul 6, 2006
    Near...
    Just subjective because I haven't done any careful measurements, but it seems like pyTivo can transfer at a higher encoding bitrate than TiVo.Net. I have two PC's networked to my TiVo via a router:

    1. 3 GHz Pentium 4, hyperthreaded, wired ethernet
    2. 2.8 GHz Pentium 4, wireless ethernet
    The TiVo is wired to the router.

    With TDN, I limited bitrates to 3 kbps on #1 and 2 kbps on #2 in order to transfer slightly greater than real-time. pyTivo always encodes at 4 kbps and it has no trouble keeping ahead of real-time on either PC.

    Anyone else get this impression?
     
  20. Mar 8, 2007 #100 of 5684
    ocntscha

    ocntscha New Member

    149
    0
    Oct 22, 2003
    Since you asked, I run Tivo.Net on a 2Ghz Athlon XP, Centos Linux. I've settled on leaving ffmpeg configured at 4096/kbs which gives me slightly faster than real time.
     

Share This Page