pyTivo - memory leak or problem with my setup?

[bacevedo]

I installed the latest version of pyTivo a couple of days ago and so far I really like it. Much faster and more reliable than Tivo Desktop. (Rant: Why after so many years and versions does Tivo Desktop still have issues like memory leaks and locking up?) Plus, being able to watch my videos from my camera and podcasts is very nice. My wife loves that!

But I noticed something about the ffmpeg_mp2.exe in Task Manager. I am running pyTivo as a service and when I listen to music (AAC files) everything works fine as long as I don't skip tracks (i.e. ch+ and ch- to go the next or previous song.)

Under normal operation, it appears that a new ffmpeg_mp2.exe process is created for each song. As soon as the current song ends, the ffmpeg_mp2.exe process ends and a new one is created for the next song. However, when I end the song abruptly, (i.e. go to the next song with a ch+), a new ffmpeg_mp2.exe process is created, but the old one stays around. I noticed this because I checked my task manager and had around 20 ffmpeg_mp2.exe processes, but only one was actively using processor resources. In addition, once I use track skipping, pyTivo won't automatically skip to the next track. It just hangs at the end of the song until I tell it to go to the next song.

So, is this normal, a known bug or is something wrong with my setup? If I use pyTivo as a console (i.e. not a service) does it act differently? I like the service set up because my family can use it to view videos, pictures and music without me there to fire up the server.

For reference I have a Tivo HD with 9.3. I am running Windows XP (SP2) with a hardwired network. All of my music is managed in iTunes and are AAC (m4a) files.

Thanks,
Bryan

P.S. One last question. How do I get pyTivo to display music information for the song that is playing? Right now it only shows the name of the file, but says unknown for everything else. I am assuming I need to create some sort of ID3 tags, but I have never had to do that before (and therefore no experience with it), so I want to make sure I know what pyTivo uses before going through the work.

[wmcbrine]

However, when I end the song abruptly, (i.e. go to the next song with a ch+), a new ffmpeg_mp2.exe process is created, but the old one stays around.It's supposed to be killed. But I've seen the lines to do that mysteriously disappear during a merge... You should clarify what you mean by "the latest version", since there are four forks, each with several branches.

It's also possible that the kill function just isn't working in Windows. It should be, but I don't know that I ever checked. (I just import it from the video plugin.)

In addition, once I use track skipping, pyTivo won't automatically skip to the next track. It just hangs at the end of the song until I tell it to go to the next song.This is something that I'd thought was fixed when we added transcoding and ff/rewinding (which is based on transcoding). Prior to that, I'd seen it a lot; I don't think I've seen it since. Anyway, it's a TiVo-side bug.

P.S. One last question. How do I get pyTivo to display music information for the song that is playing? Right now it only shows the name of the file, but says unknown for everything else. I am assuming I need to create some sort of ID3 tags, but I have never had to do that before (and therefore no experience with it), so I want to make sure I know what pyTivo uses before going through the work.ID3 tags are an MP3 thing, AFAIK. For AAC... there are some commented-out lines in music.py that derive the info from the directory layout, if you want to try them. I also have some code from David Rein (who did the original transcode support) to use "AtomicParsely" to extract tags from AAC files, which I hadn't committed because I didn't really want to introduce another dependency. But 99% of my own collection is MP3, so I'm not the best person to ask about AAC.

[bacevedo]

It's supposed to be killed. But I've seen the lines to do that mysteriously disappear during a merge... You should clarify what you mean by "the latest version", since there are four forks, each with several branches.


I am using KRKeegans' fork on the Current Release page at pytivo.armooo.net. I just downloaded the Windows Installer package and ran it (It says it is ideal for new users). Maybe that one is out of date with your fork? I believe the date on it is May 2, so it is relatively new.


This is something that I'd thought was fixed when we added transcoding and ff/rewinding (which is based on transcoding). Prior to that, I'd seen it a lot; I don't think I've seen it since. Anyway, it's a TiVo-side bug.


Yeah - I saw this with Tivo Desktop (all versions) and was hoping that it would work for me with pyTivo. Do you think changing to your fork would change this behavior?


ID3 tags are an MP3 thing, AFAIK. For AAC... there are some commented-out lines in music.py that derive the info from the directory layout, if you want to try them. I also have some code from David Rein (who did the original transcode support) to use "AtomicParsely" to extract tags from AAC files, which I hadn't committed because I didn't really want to introduce another dependency. But 99% of my own collection is MP3, so I'm not the best person to ask about AAC.

Yeah - after I posted I did some research on ID3 and noticed that it was for MP3. If I download your fork (the tar.gz file), what lines would I need to update in music.py to have it derive at least some info from the folder structure? That's really all I need (artist, album and song name). I don't care about any of the other stuff.

What do I do with your downloaded file? Just decompress it to my pyTivo install folder? Do I need to compile anything?

Also, is there something I could change to make pyTivo respect the shuffle setting on my Tivo HD for playlists?

Thanks again!

Bryan

[wmcbrine]

Maybe that one is out of date with your fork?No.

Yeah - I saw this with Tivo Desktop (all versions) and was hoping that it would work for me with pyTivo.No. The fix I was talking about was when going from not supporting ff/rewind (old versions of pyTivo) to supporting it (new versions of pyTivo). But TiVo Desktop has always supported it, AFAIK. So if you see the problem with TD, you're not likely to fix it with pyTivo, sorry.

Do you think changing to your fork would change this behavior?No.

what lines would I need to update in music.py to have it derive at least some info from the folder structure?Strip the leading '#' from these lines:

#artist, album, track = f.name.split(os.path.sep)[-3:]
#track = os.path.splitext(track)[0]
#if track[0].isdigit:
# track = ' '.join(track.split(' ')[1:])

#item['SongTitle'] = track
#item['AlbumTitle'] = album
#item['ArtistName'] = artist


What do I do with your downloaded file? Just decompress it to my pyTivo install folder?That's all you'd need to do, yes. But I don't recommend you do it at this time.

[Phantom Gremlin]

(Rant: Why after so many years and versions does Tivo Desktop still have issues like memory leaks and locking up?)
Your rant was perhaps rhetorical. But I'll take the bait anyway.

TiVo outsources their engineering. How much of it, I don't know. But if TiVo Desktop was written by Apu in Bengaluru it's because the company Apu works for was the low bidder in a global "race to the bottom". In the final analysis, Apu doesn't really give a f*** about your problems.

On the other hand, you're getting 1-on-1 support here from wmcbrine. I don't know why he works on this sort of "underground" software. But I'd venture to guess it's because he is passionate about what he's doing.

Can wmcbrine help you solve your problems? I don't know, but I'd bet on him before I bet on Apu. And, to forestall all the posts telling me I'm being xenophobic, I'm not putting down Apu's abilities (for all I know, wmcbrine is as Indian as Apu). I'm putting down the results of all of these outsourced engineering efforts. The more a company outsources key facets of it's business, the worse the results are.

[bacevedo]

wmcbrine - thanks for your help and info. I uncommented out the code above and it works great! The only issue is when the albums are in the "Compilation" folder instead of the artist folder (like a greatest hits album). Then the artist says compilation. I might change that in iTunes so that those albums go back to their artist folder anyway. Only soundtracks would be screwed up (multiple artists), but that's fine - it's better than nothing!

I was playing with the music.py this weekend and was spitting out some debug messages to the console to see the flow of the program.

My question for you is where does the ffmpeg process get killed? I see in music.py where you kill it, but that's only if an exception occurs when writing the file out to the stream. How is it supposed to get killed when the next song starts up? Does ffmpeg just exit automatically after it finishes transcoding the file? (That is what appears to be happening - there is nothing that kills ffmpeg when it is interrupted and it doesn't gracefully shut itself down.)

I just need to know where the "hook" is when I press ch+ or ch-. I was trying to capture the process id in a global variable, and then when ffmpeg is going to be fired again, I was going to kill the previous process id. However, I don't know python and am not sure how the scoping of the variables work. It wasn't keeping my values - I am guessing music.py is called anew for each song request. Where would I define a global variable that can capture the process id for the current ffmpeg process. Then right before ffmpeg is called, I kill the process id for the previous ffmpeg (if it exists).

Thanks,
Bryan

[wmcbrine]

My question for you is where does the ffmpeg process get killed? I see in music.py where you kill it, but that's only if an exception occurs when writing the file out to the stream. How is it supposed to get killed when the next song starts up?That's it -- when you tell the TiVo to skip, it closes the socket, which causes an exception.

Does ffmpeg just exit automatically after it finishes transcoding the file?In normal playback, with no manual skipping, yes, that's how it works. It reads in the file and writes out the transcoded version. No reason for it not to exit then.

[bacevedo]

That's it -- when you tell the TiVo to skip, it closes the socket, which causes an exception.


Interesting. When I was writing out print messages in the "except" part of the code, they weren't always getting written out. It's like the exception never occured when I pressed ch+. Also, I wrote messages outside of the "except" block (i.e. they would print if there was no exception), and they seemed to be printed out while ffmpeg was still transcoding. So it looks like music.py doesn't sit there and wait for ffmpeg to return or should it? It appears to me that ffmpeg is acting like its own thread and the process flow continues even while the transcoding is occuring. I'll have to verify that, though, because there was a lot printed to the screen and I may be remembering it wrong.

Bryan

[bacevedo]

OK - I did some more testing with it last night. I added a finally: block to see if that would be called when a ch+ was pressed. It didn't get called.

What appears to be happening on my Windows system is that ffmpeg does not throw an exception when ch+ is pressed. It just kind of hangs there and music.py does not get control back. It appears to be more of an issue with ffmpeg or my Tivo HD not closing the socket instead of music.py having an issue. I probably will do some more playing with it and to see how the ffmpeg tool works and to see why it may not throw an exception when the socket is closed.

As for my comments earlier about music.py not waiting - you can ignore that. It appears to be waiting (blocking) correctly when the file stream is being written out by ffmpeg.

Bryan