1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pyTivo - Transcoding server

Discussion in 'TiVo Home Media Features & TiVoToGo' started by armooo, Nov 25, 2006.

  1. Mar 2, 2007 #61 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Yeah pyTivo checks for ffmpeg compatibility the first time it opens a folder, it is then cached so it should be fast. TiVo.Net checks for compatibility when it loads. I can't decide what is preferable.
     
  2. Mar 3, 2007 #62 of 5687
    armooo

    armooo pyTivo Developer

    81
    0
    Feb 1, 2003
    You are going to need python 2.4. I am using the subprocess module which was introduced in 2.4.
     
  3. Mar 3, 2007 #63 of 5687
    armooo

    armooo pyTivo Developer

    81
    0
    Feb 1, 2003
    I prefer the scanning the first time a folder is opened for a few a few reasons.

    1) Ever time you start your computer do you want to to scan all your files?

    2) Do you want to keep the status of every file in memory? Right now it will only keep the 100 most recently seen.

    3) Finding an item in the tivo menu gets painful when there are more the 40 items. I just opened a folder with 60 videos and it took about 5 sec. And I sort my videos in to folders and sub folder so I never have more than 22 in a folder.
     
  4. Mar 3, 2007 #64 of 5687
    dlfl

    dlfl Cranky old novice

    7,004
    19
    Jul 6, 2006
    Near...
    Hmmm..... Doesn't seem to work :confused:

    Using Windows XP SP2. Tivo.Net and Desktop and goback all work fine on this machine. Downloaded and installed Python 2.5 and pyTivo Ver. 165. No pyTivo ("video" ??) shows up on the NPL.

    Here is my .conf file:

    [Server]
    port=9032
    ffmpeg=C:\Program Files\pyTivo\plugins\video\ffmpeg_mp2.exe

    [<Name of Share>]
    type=video
    path=C:\Video\pyTivoVideos

    I don't use Windows firewall. I have a hub but it isn't keeping tivo.net from running. I get the "Cheetah" warning at startup but that's not supposed to be a problem per prior posts.

    Is the desktop TiVo server supposed to be running or not?

    Any ideas, please ?

    EDIT:
    Realized I needed to put a name in instead of <Name of Share> and that got it working to the point of showing up on the NPL.

    However the first two files I tried to transfer failed with ffmpeg errors. Here is an example of pyTivo response for one of them:

    192.168.0.153 - - [03/Mar/2007 10:17:54] "GET /pyTivoShare/CleeseMartinOutTowner
    s.mp4 HTTP/1.1" 200 -
    ffmpeg version 0.4.9-pre1, build 4747, Copyright (c) 2000-2004 Fabrice Bellard
    configuration: --enable-memalign-hack --enable-mingw32 --enable-mp3lame --ena
    ble-gpl --enable-a52
    built on Mar 21 2005 16:41:12, gcc: 3.4.2 (mingw-special)
    Input #0, mov,mp4,m4a,3gp,3g2, from 'C:\Video\pyTivoVideos/CleeseMartinOutTowner
    s.mp4':
    Duration: 00:03:52.8, start: 0.000000, bitrate: 596 kb/s
    Stream #0.0 Id: 1: Video: h264, 320x240, 29.97 fps
    Stream #0.1 Id: 2: Audio: 0x6134706d, 48000 Hz, stereo
    Output #0, vob, to 'pipe:':
    Stream #0.0 Id: 0: Video: mpeg2video, 720x480, 29.97 fps, q=2-31, 4096 kb/s

    Stream #0.1 Id: 1: Audio: mp2, 48000 Hz, stereo, 192 kb/s
    Stream mapping:
    Stream #0.0 -> #0.0
    Stream #0.1 -> #0.1
    Unsupported codec (id=86018) for input stream #0.1

    Can't imagine it would actually fail on mp2 audio :confused:

    The other file was a .wmv file and it complained about top crop or something like that. Both files transfer OK with Tivo.Net.

    Seems like a pretty old build of ffmpeg (??). The one that comes with Tivo.Net is about twice the file size (??)

    Hah !! Substituted the ffmpeg_mp2.exe that comes with Tivo.Net and it worked, although the video was blocky compared to transfering with Tivo.Net. Looks like the encoding bitrate is too low -- any way to adjust that?
     
  5. Mar 3, 2007 #65 of 5687
    ocntscha

    ocntscha New Member

    149
    0
    Oct 22, 2003
    Doh! Ok, I'll check into making that happen. I did try it with Python 2.3.4 though, it seemed to at least start up but never showed in my NPL. I fired up tcpdump and it appears pyTivo is broadcasting out my public ip address and not the LAN one. I had the exact same problem with Tivo.Net until Pipakin addeded the option of picking which ip address to bind to.
     
  6. Mar 3, 2007 #66 of 5687
    windracer

    windracer joined the 10k club

    11,580
    3
    Jan 3, 2003
    St. Pete, FL
    I originally had Python 2.3 on my Mandrake box and pyTiVo wouldn't run either, so I upgraded to 2.5.
     
  7. Mar 3, 2007 #67 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Hmm okay let me check out Pipakin's windows ffmpeg binary. If it is an updated version we can replace ours.

    Choppy video?? Hmm i don't have a setting to change the bit rate, but I would be surprised if that was the issue. Currently the bit rates are hard coded at 4096kbits/second video and 192kbits/second audio.

    I will look into adding this as an option the config file.

    Hmm okay, let me see if I can add an ip as an option in the config file

    Feature Request
    Ok so it would be nice to add metadata. But I don't know what is the best way to handle this:
    1. I would love to use embedded data in the video file itself such as RIFF data. This would be ideal, any changes to the file would not destroy the metadata. However I dont think there is a good python module for doing this without requiring a major install.

    2. Central DB. This is still very clean, but changing the name or moving an video files would delete the metadata. Additionally the loss or damage to the main DB would mean the loss of ALL data.

    3. Individual files. Messy ugly and prone to create excess files. Not really an option.

    Let me know what you think
     
  8. Mar 3, 2007 #68 of 5687
    Ron667

    Ron667 New Member

    10
    0
    Feb 8, 2007
    I believe that the newer version of ffmpeg that is packaged with Tivo.net changed the way the bitrate was defined. In the old version, you specified in kbytes, in the new version you specify in bytes. I seem to remember when Pipakin switched over and had this same problem of the resolution suddenly going through the floor.

    Sorry I don't have a link to that post in the other thread.
     
  9. Mar 3, 2007 #69 of 5687
    dlfl

    dlfl Cranky old novice

    7,004
    19
    Jul 6, 2006
    Near...
    Yep, in the newer version you have to use "4000k" instead of "4000" (for example). I'm going to look at transcode .py to see if there is an obvious place to insert the 'k'.

    Edit:
    Found it, line 23 of transcode.py:
    cmd = [FFMPEG, '-i', inFile, '-vcodec', 'mpeg2video', '-r', '29.97', '-b', '4096K'] + select_aspect(inFile) + ['-comment', 'pyTivo.py', '-ac', '2', '-ab', '192', '-f', 'vob', '-' ]

    Note the 'k' added to 4096.

    This worked fine for my .mp4 (H264 video, mp2 audio) file.

    Thus just copy a better version of ffmpeg_mp2.exe (e.g., the one that comes with Tivo.Net) in to replace the original, and make this simple edit to transcode.py and you are in business. (Warning: have not tried it as a service yet.)
     
  10. Mar 3, 2007 #70 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Yup that will do it, and it won't have any bad effects when running as a service.

    Edit Ok adding K to the older version of ffmpeg has no negative effects. So I will change this in the release tonight. Other things in the list for tonights release: One final tweak to the transfer error.
     
  11. Mar 3, 2007 #71 of 5687
    dlfl

    dlfl Cranky old novice

    7,004
    19
    Jul 6, 2006
    Near...
    I'm having trouble with several files I transfer that are 1 to 5 minutes long. They transfer and view fine and ask if you want to save them. However when you try to play them from the NPL they cannot be played and it says TiVo was unable to record them perhaps because the signal was lost. (I sse this sometimes with TiVo.Net too)

    Here is an excerpt from the pyTivo command window:

    frame= 8997 q=1.6 Lsize= 118824kB time=300.2 bitrate=3242.9kbits/s
    video:110261kB audio:7038kB global headers:0kB muxing overhead 1.300029%
    192.168.0.153 - - [03/Mar/2007 21:59:56] "GET /pyTivoShare/CivWarPhotogNorth5min
    Fix.wmv HTTP/1.1" 206 -
    192.168.0.153 - - [03/Mar/2007 22:01:16] "GET /TiVoConnect?Command=QueryContaine
    r&Container=pyTivoShare&Recurse=Yes&SortOrder=!CaptureDate&ItemCount=8&Filter=x-
    tivo-container%2Ftivo-videos,x-tivo-container%2Ffolder,video%2Fx-tivo-mpeg,video
    %2F* HTTP/1.1" 200 -

    This excerpt starts with the final 2 output lines from ffmpeg when it finished encoding. The time it gives is exactly correct for the clip. Notice the "GET ....... 206" that comes next. Is that a clue as to the problem?

    Also, you posted on the Tivo.Net thread that the solution to end of transfer involved sending TiVo a 206 with certain headers. I don't see those headers in the GET....206 command, but I don't know if they should be visible there.

    You may notice the bitrate is well below 4096k. That's partly because I had changed it to 3072k in transcode.py as an experiment to see how it affected this problem. The behavior was the same when it was at 4096k. The WMV file I'm transfering is highly compressed to around 600 kbps, and I don't think ffmpeg can meet the high bitrate targets. (Perhaps that is part of the problem?)
     
  12. Mar 3, 2007 #72 of 5687
    dsandor

    dsandor New Member

    2
    0
    Jan 19, 2007
    When I post this url from a browser on the pyTivo machine I get this error:

    h.t.t.p:./. / 127.0.0.1:9032 /TiVoConnect?Command=QueryContainer&Container=Video

    generates:

    C:\Program Files (x86)\pyTivo\Cheetah\Compiler.py:1508: UserWarning:
    You don't have the C version of NameMapper installed! I'm disabling Cheetah's useStackFrames option
    as it is painfully slow with the Python version of NameMapper. You should get a copy of Cheetah with
    the compiled C version of NameMapper.
    "\nYou don't have the C version of NameMapper installed! "
    stewie - - [03/Mar/2007 22:56:33] "GET / H.T.T.P/1.1" 200 -
    stewie - - [03/Mar/2007 22:56:33] "GET /favicon.ico H.T.T.P./1.1" 200 -
    192.168.210.234 - - [03/Mar/2007 22:56:40] "GET /TiVoConnect?Command=QueryContainer&Container=%2F H.T.T.P/1.0" 200 -
    ----------------------------------------
    Exception happened during processing of request from ('192.168.210.234', 2381)
    Traceback (most recent call last):
    File "C:\Program Files (x86)\Python\lib\SocketServer.py", line 463, in process_request_thread
    self.finish_request(request, client_address)
    File "C:\Program Files (x86)\Python\lib\SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    File "C:\Program Files (x86)\Python\lib\SocketServer.py", line 521, in __init__
    self.handle()
    File "C:\Program Files (x86)\Python\lib\BaseH.T.TPServer.py", line 316, in handle
    self.handle_one_request()
    File "C:\Program Files (x86)\Python\lib\BaseH.T.TPServer.py", line 310, in handle_one_request
    method()
    File "C:\Program Files (x86)\pyTivo\h.t.tpserver.py", line 49, in do_GET
    self.RootContiner()
    File "C:\Program Files (x86)\pyTivo\h.t.tpserver.py", line 73, in RootContiner
    self.wfile.write(t)
    File "C:\Program Files (x86)\Python\lib\socket.py", line 261, in write
    self.flush()
    File "C:\Program Files (x86)\Python\lib\socket.py", line 248, in flush
    self._sock.sendall(buffer)
    error: (10053, 'Software caused connection abort')
    ----------------------------------------
    192.168.210.234 - - [03/Mar/2007 22:57:39] "GET /TiVoConnect?Command=QueryContainer&Container=%2F H.T.T.P/1.0" 200 -
    192.168.210.234 - - [03/Mar/2007 22:58:39] "GET /TiVoConnect?Command=QueryContainer&Container=%2F H.T.T.P/1.0" 200 -
    192.168.210.234 - - [03/Mar/2007 22:59:39] "GET /TiVoConnect?Command=QueryContainer&Container=%2F H.T.T.P/1.0" 200 -
     
  13. Mar 4, 2007 #73 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Hmm I not sure what is causing this. But I noticed that Desktop sent 3 hexcodes at the end of a transfer. Adding this seems to help, I think maybe they mean end of file??

    Anyways try the new version, I hope that it will solve your problem.
    Nope that is exactly how it should work. The "GET" command comes from TiVo requesting more of the file that isnt there. The 206 is our response back to TiVo the headers are sent but not printed in the output.

    Hmm, i am not sure what you mean here. ffmpeg handles fine for me at 4096K. I am able to transcode about 30% faster than realtime. But this is on linux and using a 2.8GHz processor. Lower bitrates should run better on slower machines.

    NEW VERSION
    Just minor changes:
    - Added some final code to end transfers properly
    - Changed code to allow for use of newer version of ffmpeg

    I didn't switch to pipakin's ffmpeg because it is pretty large. In fact I think we should pull ffmpeg out of the distro so people dont have to download it for every new version. Armooo what do you think?
     
  14. Mar 4, 2007 #74 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    dsander, could you describe to me more of your problem.

    First is pyTivo working at all?? What XML comes up when you use this URL

    :9032/TiVoConnect?Command=QueryContainer&Container=

    And we will work from there
     
  15. Mar 4, 2007 #75 of 5687
    dlfl

    dlfl Cranky old novice

    7,004
    19
    Jul 6, 2006
    Near...
    Three WMV files (1, 2.5 and 5 mins long) that would transfer but not play later from the NPL using 165 now both transfer and play later OK with 166.

    Thanks KRKeegan and Armooo :up:

    The only slight glitch was at the end of transfering the 1 min clip. It stopped and gave the wait-and-hit-play message. I had to do that several times, taking perhaps 20 seconds before it finished up in a normal way. This is not a result of slow video serving from my PC -- it was well ahead of real-time at all times during the transfer. It must be a file-length issue, I would think.

    @KRKeegan:
    I'll try to clarify what I was trying to say before. Normally with a 4096k bitrate specified, ffmpeg will encode at that rate plus a few 100 kbps for the audio. However for very low bitrate input files (such as my WMV's) it can't see its way clear to do this and ends up encoding with a smaller bitrate. I assume you are basing the file length estimates you send to TiVo on the specified bitrate, but the lower actual rate results in a smaller actual file size. It would be plausible to me for this to make end-of-transfer problems more likely. (???)

    I am still using the ffmpeg from TiVo.Net just FYI.

    I notice a delayed response to clicks when doing things in the pyTivo section of the NPL and it looks like messages are occuring about every ten seconds in the pyTivo command window. Is this timing set by pyTivo or by the TiVo itself?

    Looking good so far -- thanks again :)
     
  16. Mar 4, 2007 #76 of 5687
    Zothar

    Zothar New Member

    2
    0
    Mar 4, 2007
    Great program guys. I had to do a few tweaks to get things to work for me and I hope they're useful for others as well.

    For one video, which as 720x304 IIRC, the select_aspect_ratio() function was coming up with negative padding values, which my ffmpeg complains about. The changes are in svn diff format in the attached aspect_ratio_change patch file.

    My ffmpeg's output is formatted slightly differently, so I made a change so both the old and new formats should work. The changes are in svn diff format in the attached ffmpeg_change patch file.

    I've noticed that ffmpeg seems to be called several times to get various bits of info about the video file before transcoding. I'm thinking about making a generalized file info mechanism using the LRU caching mechanism, but haven't coded it up yet as it'll be a refactor compared to the above mentioned changes.

    BTW, setting the svn:eol-style property to "native" on the various files in Subversion would make things easier for me and perhaps others since I'm on a Debian Linux box and it appears others may be using Windows.
     

    Attached Files:

  17. Mar 4, 2007 #77 of 5687
    Jabo4

    Jabo4 New Member

    21
    0
    Feb 17, 2007
    Cool great updates. Couple questions/issues.
    1. I know you said you weren't sure which method you liked on scanning the files, I much prefer populating that list as fast as possible on the tv as opposed to the current way. Is there a way to put both methods in code and give us a setting for that?

    2. Windows service still doesn't work, I am unfamiliar with Python so I have no clue how to "run" that without just clicking on it. The screen pops by too fast to read the error. It is in my services, but it just doesn't start properly. How can I get this error logged so I can get a better look at what might be causing it?

    But thanks! the transfers are working flawlessly for me since the updates as well.
     
  18. Mar 4, 2007 #78 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    Yeah this is a function of how TiVo requests the file. The process works like this.
    1. TiVo asks for the information about a video
    2. pyTivo sends back a file size that is as large as the file could possibly be (Example 1Gig)
    3. TiVo sends a get request for the Video
    4. pyTivo streams the video as it transcodes
    5. Stream abrubtly ends at the end of the video (Example 800MB)
    NOTE: At this point TiVo is still expecting another 200MB
    6. TiVo waits thinking that network traffic has slowed down the packets, TiVo seems to wait 30 seconds or so.
    7. TiVo then requests the file again asking for the last 200MB
    8. pyTivo responds with a 206 which tells TiVo that the file has ended and sends a short bit of code which I believe equals and End of File line.

    The problem occurs in steps 6-8. Until TiVo receives the 206 response it leaves the file open expecting more to come. I don't think I can force TiVo to respond any faster. And I dont think i can just send the 206 until TiVo requests it. As for bad estimations in the file size. They dont really matter, too large just clears more space than necessary but it doesnt affect the delay at all in steps 6-8. Too small clearly would cause the video to end prematurely, so I would rather err on the much to large side than be super accurate.

    I think for very short files we may always have small problems trying to play them live. I will look into this but since playing files 1 minute in length is rare I won't consider it critical.

    Hmm, the TiVo UI is a little slow, but 10 seconds sounds excessive. 10 seconds may occur the first time you load a folder after starting pyTivo. But if you exit the NPL and come back that folder should load in seconds because the info is in the cache the second time.

    Oops I thought I fixed all of these issues. Thanks for the help

    Again thanks for the help.

    Hmm, I think(??) that ffmpeg is only called at most twice per file. Once to get the video info(Size, Duration, Type) which is then cached in LRU. And a second time to transcode the file. If there are places where it is calling ffmpeg unnecessarily please let me know, but I think we caught them all.

    I not in charge of SVN, and I dont even know what that is so maybe armooo can fix that for you.

    As for your updates, since they are not critical I will toss them into a cumulative update tonight.
     
  19. Mar 4, 2007 #79 of 5687
    KRKeegan

    KRKeegan Im lost and confused

    215
    0
    Jul 20, 2004
    Los Angeles, CA
    teehee, you snuck in there while I was writing my previous message.
    Yeah I was thinking about that too. I think this is doable.

    Hmm, the windows service has been armooo's teritory.

    But to run it so you can see the errors:
    1. Start->Run
    2. Type "cmd"
    3. browse to the folder where pyTivo is located. For example "cd c:\program files\pytivo"
    4. Type "pyTivoService.py"

    That will run the service in the command window, all errors will be displayed there. If you need to end a python program it can be accomplished by presseing CTRL-BREAK
     
  20. Mar 4, 2007 #80 of 5687
    dlfl

    dlfl Cranky old novice

    7,004
    19
    Jul 6, 2006
    Near...
    I just transfered a 5 minute .mpg clip that was edited from a TiVo recording using VideoReDo. I thought it should have transfered without re-encoding, as stated in the first post of this thread:
    However, not only did it start re-encoding, but the aspect ratio on the TV was wrong (corresponding to the 480x480 format of the file). :confused:

    This same clip will go back using TiVo go back and plays with the correct AR. The VideoReDo info says the display resolution is 720x480 (but encoding resolution is 480x480), as it does for all TiVo recordings or edited versions thereof.
     

Share This Page