1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

PyTivo Video Manager HME App for pyhme

Discussion in 'Developers Corner' started by jbernardis, Apr 20, 2011.

  1. orangeboy

    orangeboy yes, I AM orangeboy!

    4,083
    0
    Apr 19, 2004
    East Moline, IL
    As does pyHME... ;)
     
  2. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    Do you see anything on the TV before this message? What is in the directory you are currently in? Can you give me a dierctory listing?
     
  3. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    I have a new version out on git that addressed some of the concerns of lrhorer. This is version 0.5d. The changes are:
    Version 0.5d 6/14/11
    - added logic to load thumbnails just in time rather than ahead of time. This speeds up initial painting of a page, but slows down navigation within the page. Beacuse of this,
    - I added an image cache to speed up image processing. This is just an in-memory cache, but if you exit the app cleanly - by hitting left all the way out - the cache is saved to disk for future runs. The cache quickly got too big to save to disk with every change, so this was a compromise
    - added logic to scale the thumbnail to fit in the 320 pixel wide view frame. Thanks to wmcbrine for the picture sample hme app that told me how to do this. A side effect of this - you need to have the python image lib (PIL) installed. Just google PIL and you will find links for the version of python you have installed.

    I am a bit concerned over the size the cache can grow to - so I'd be interested if lrhorer could give me his experience with this. I might have to implement something that limits the size of the cache.

    I could also provide a tool to generate the cache ahead of time if there is interest in that. Right now, there is a little sluggishness the first time you load an image, but once it's in the cache, things are snappier. If you pre-create the cache, you avoid this first time pain.


    NOTE there is a new py file (thumbcache.py) in the vidmgr directory.
     
  4. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    What, the TSNs? Are those sensitive?
     
  5. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    It's not on the git, yet. Did you upload it? Also, the download is still not working here in FireFox. It works for IceWeasel.
     
  6. wmcbrine

    wmcbrine Ziphead

    10,368
    22
    Aug 2, 2003
    pyTivo uses the lrucache module for this purpose.
     
  7. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    Sorry about this everybody - I guess I was tired last night. Apparently I forgot to upload to git. I THOUGHT I had done so. I am at work and do not have the latest code with me - it will have to wait until this evening.
     
  8. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    I was thinking of some kind of LRU algorithm, but I didn't realize python has one already. I need to explore the modules more.

    I also need to explore how to make the cache threadsafe. Since the cache exists at the module level, there could potentially be multiple threads using it. I assume python has some kind of mutex mechanism.
     
  9. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    rrr22777 - I'm just looking a bit closer at the code, and I think I know what happened. I just want to make sure my assumptions are accurate:

    Since it appears you are new to pytivo also, I guessed that 1) you have no videos in your shares, and 2) you only have a single share.

    I think I see where the logic falls apart in that case. In that situation, there is basically nothing for the app to do - there is nothing to navigate. I can test that tonight - I already know how to fix it, and if I am right, I will post the fix with the release I apparently forgot to upload last night.
     
  10. orangeboy

    orangeboy yes, I AM orangeboy!

    4,083
    0
    Apr 19, 2004
    East Moline, IL
    Hmm... Good question. But yes, that's what I was referring to. There's been some discussion about disclosing one's TSN publicly, but really no examples of what another could do with such a TSN.

    Apologies for meandering off-topic.
     
  11. rrr22777

    rrr22777 New Member

    532
    0
    Jul 31, 2002
    Long time pytivo user. Also use streambaby but pytivo never crashes while steambaby does on some videos. Both are running on the same computer with the same ffmpeg binary.

    As for pytivo only 1 video share but about 50 folders in that share with videos in various formats. Gives me an error saying it cannot transfer on all videos but only crashes on some foldesr. I am mostly looking into this app for the delete functionality. Can the ability to just initiate a regular download to tivo be added if the video does to conform to the right standard?

    I can upload more debug info when I get home if you would like. 100% with the poster that said Tivo needs to learn a few things about how to display its NPL based on this app.
     
  12. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    It's interesting that it only crashes on some folders. This app does not process the files at all - it makes push requests to pytivo through its HTTP interface. All my app does, basically, is process directories and read some metadata information.

    Whatever info you can get to me would be beneficial - particularly the contents of the directories that crash. I think for some reason my app thinks you have an empty directory but yet it is continuing to process it. I can't figure out why.

    Also, do you know what version you are running? I just added a print statement at startup that displays the version number, but this wasn't in early versions. The version number is up near the top of the __init__.py file.
     
  13. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    This time I made certain of my posting - version 0.5d is definitely on git. It's probably best that this was delayed a day because I was able to address the concerns I mentioned yesterday. I now use LRU logic to maintain the last X images in memory. X is initially set to 100. It's currently not configurable because the cache is initialize at module import time and "normal" access to the config file is unavailable. I can parse the file myself if we find we need to be able to tune this parameter.

    I limited it to 100 because I was concerned about memory use - especially on my system (which is my readynas server). Since the cache is maintained at the module level, I needed to add mutex protection; however, this also means that the cache will be maintained from one usage to the next. In addition to this, if you exit the app cleanly - by hitting left all the way out, the cache will be written to disk - so it will survive a stop/restart of the server (although the lru ordering will not survive a restart)

    Overall, I'm not 100% on board with the cache. I think we gained sufficient performance gain my loading the images just in time rather that in advance. I think the marginal gain realized by the cache might not be worth the processing required. But again, we'll see how it goes.

    rr22777:

    I definitely found a hole in the logic that would have caused the exact problem you saw. Frankly I'm surprised it hasn't surfaced before now. Try this version and see if it solves your problem.

    Version 0.5d 6/15/11
    - added logic to load thumbnails just in time rather that ahead of time. This speeds up initial painting
    of a page, but slows down navigation within the page. Beacuse of this,
    - I added an image cache. If you exit the app cleanly - by hitting left all the way out - the cache is saved to disk
    for future runs. The cache got too big to save to disk with every change, so this was a compromise. The cache uses lru
    logic to keep the last X used images in memory. I had to choose some kind of limit here - I chose 100 for now - we'll
    see how that works out. Also, the cache exists at the module level, but yet is accessed, and possible changed, at the
    HME thread level. Because of this, I had to mutex exclusion around parts of the cache logic.
    - added logic to scale the thumbnail to fit in the 320 pixel wide view frame. Thanks to wmcbrine for the picture
    sample hme app that told me how to do this. Side effect of this - you need to have the python image lib (PIL) installed
    - widened the right hand margin on the info page a bit
     
  14. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    Ooh. That's definitely much, much faster ( ~3 seconds vs. 39 seconds on an 800+ video share ), and the cache code seems to be working just fine, but the PIL code has introduced a bug. A large number of the .jpg files in my shares are still 0 bytes in length. (They are place-holders.) Whenever I try to select one of the videos whose thumbnail is 0 bytes in length, vidmgr locks up. I can exit the app in the TiVo by pressing the <TiVo> button (the left arrow does not work), but if I enter vidmgr again, it is still locked up. I have to restart pyHME to get it working again. I'm still working on getting thumbnails for all my videos, but at this point in time, nearly half are still 0 bytes.

    Code:
    Traceback (most recent call last):
      File "/usr/lib/python2.6/SocketServer.py", line 560, in process_request_thread
        self.finish_request(request, client_address)
      File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
        self.RequestHandlerClass(request, client_address, self)
      File "/usr/share/pyhme/start.py", line 141, in __init__
        client_address, server)
      File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
        self.handle()
      File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
        self.handle_one_request()
      File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
        method()
      File "/usr/share/pyhme/start.py", line 232, in do_GET
        self._page(True)
      File "/usr/share/pyhme/start.py", line 186, in _page
        appinst.mainloop()
      File "/usr/share/pyhme/hme.py", line 1134, in mainloop
        while self.active and self.get_event():
      File "/usr/share/pyhme/hme.py", line 1187, in get_event
        handle(keynum, rawcode)
      File "/usr/share/pyhme/vidmgr/__init__.py", line 322, in handle_key_press
        self.handle_key_pressList(keynum, rawcode)
      File "/usr/share/pyhme/vidmgr/__init__.py", line 469, in handle_key_pressList
        self.drawScreen()
      File "/usr/share/pyhme/vidmgr/__init__.py", line 768, in drawScreen
        self.drawScreenDetail()
      File "/usr/share/pyhme/vidmgr/__init__.py", line 942, in drawScreenDetail
        self.listing[self.indexDetail]['name'])
      File "/usr/share/pyhme/vidmgr/__init__.py", line 1424, in getThumb
        data = tc.getImageData(tfn)
      File "/usr/share/pyhme/vidmgr/thumbcache.py", line 129, in getImageData
        self.cache[filename]['data'] = resizePic(filename, self.width, self.height)
      File "/usr/share/pyhme/vidmgr/thumbcache.py", line 17, in resizePic
        pic = img.open(fn)
      File "/usr/lib/python2.6/dist-packages/PIL/Image.py", line 1980, in open
        raise IOError("cannot identify image file")
    IOError: cannot identify image file
     
  15. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    I can, and probably should, put a try block around that code. I'll get something out right away.

    Great to hear the other changes are for the better.
     
  16. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    You weren't clear in your e-mails why increasing the width of the view frame would be an issue, but I would definitely still like to lobby for a wider view frame, if it is at all a reasonable thing to implement in code. In my opinion, there is absolutely no question that this

    [​IMG]

    and this

    [​IMG]

    are far more impressive on-screen than this

    [​IMG]

    and this

    [​IMG]

    There's plenty of screen real-estate to support these pictures.
     
  17. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    Thanks! I suspect non-zero length, but still invalid, files might also cause the problem.

    Oh, they are, they are. You rock, man!
     
  18. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    I don't dispute the quality of the pics, and it's really simple code to have a different size image. It's just a question of layout.

    Right now, the entire left half of the screen is devoted to the various lists, the right half for details. The top of the right is for description, and the area to the right of the image is for the sub menu (push/delete) and the tivo menu.

    In the meantime, 0.5e is on git with the try block
     
  19. jbernardis

    jbernardis New Member

    1,072
    0
    Oct 21, 2003
    Princeton NJ
    Would people rather have a landscape oriented view for thumbnails as opposed to the portrait one I have? I chose the present orientation because it is a good aspect ratio for DVD cover art, but I also know that people have been using video thumbnail maker, which would lend itself to landscape orientation. I'm willing to lay the screen out with a landscape orientation if that seems to be the concensus among users. Let me know.
     
  20. lrhorer

    lrhorer Active Member

    6,924
    0
    Aug 31, 2003
    San...
    It works just great. Thanks, yet again.
     

Share This Page