Minis taking down my home LAN

Discussion in 'TiVo Mini' started by Kremlar, Sep 16, 2019.

  1. Kremlar

    Kremlar Member

    264
    4
    Jan 22, 2009
    I've been troubleshooting this over the past few months but have had no luck so far other than narrowing the cause down to my TiVo Minis.

    Intermittently my LAN will become unusable. Internet connectivity stops and access to devices on the LAN stops as well. Through trial and error I've found the issue is caused by my TiVo Minis. If I unplug devices one at a time from my network switch until things become usable again, it's always a TiVo Mini that is the culprit.

    Originally I thought it was just a freak occurrence, then perhaps one of my Minis was just having a hardware issue, but since then I've found that it happens with multiple Minis - it's not always the same one causing the trouble. Pulling the network cable from the Mini brings everything back to normal. Power cycling the mini also brings things back to normal. Until the next time.

    This only started happening I'd say around 6 months ago. Most of my Minis are older (pre-Bolt), but I have 1 or maybe 2 that are newer. So far it's been the older Minis, never a newer one causing the issue. My switch is an older Netgear 48-port gigabit that I've had forever. I'm not sure what triggered this, but it has become quite annoying.

    Before I blindly try replacing this expensive switch, has anyone else experienced anything similar and/or have a fix?

    Thanks!
     
  2. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    I have experienced the exact same thing, there is another thread if you search where there is good information. The gist of it is that if you use DHCP reservations for your TiVo mini, it most likely will resolve the issue. Seems like they are broadcast storming the network and or traffic is looping causing the issue. I dealt with this for a while before figuring out that reserving IPs on my router seems to have resolved the issue for me. I had a Sony Bluray player that would do the same thing also. I don't think they test as well as they should and so this is a side effect of poor QA at TiVo. This never happened on my Bolt Mini or Roamio units, only Mini V1.

    Good luck.

    Jared

    Sent from my SM-G965U using Tapatalk
     
    krkaufman likes this.
  3. Kremlar

    Kremlar Member

    264
    4
    Jan 22, 2009
    krkaufman likes this.
  4. Oct 9, 2019 #4 of 40
    CoxInPHX

    CoxInPHX COX Communications

    2,474
    66
    Jan 14, 2011
    Phoenix, AZ
    Just happened to me, again. I completely forgot about this issue, it happened about a year ago.
    EDIT: No wonder I didn't remember its been 2.5 years: YouTube app crashes

    My Tivos haven't had a network connection since Friday, I was just about to purchase a new router when I decided to check my Minis that rarely get used, and sure enough one of them was completely frozen. Rebooting the Mini solved the network issue.
     
    Last edited: Oct 9, 2019
    krkaufman likes this.
  5. Oct 9, 2019 #5 of 40
    Kremlar

    Kremlar Member

    264
    4
    Jan 22, 2009
    No crash since I set the DHCP reservation, but not convinced yet.
     
  6. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    I am the person who started the other thread. For what it is worth, it only happens once in a while - sometimes months apart - possibly related to fact that this mini is not frequently used. Last two times it happened, I was able to link it to someone watching YouTube earlier that day, and at least a few people in my house have a tendency of not hitting "Tivo" when done watching, leaving whatever it is playing active. I strongly suspect it is a combination of older Minis and YouTube app - though I have no idea why this would be happening. DHCP thing sounds almost plausible, but not really - it only makes sense if you have other problems on your network....

    Do you use YouTube when this happens?
     
  7. Kremlar

    Kremlar Member

    264
    4
    Jan 22, 2009
    Not related to YouTube on my side, that I'm aware of.
     
  8. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    Damn, there goes that theory. Do you get the black screen of doom on it by any chance?
     
  9. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    Yes, we use YouTube when this happens and believe it or not DHCP and this issue in my house are directly related, I can reproduce it.

    Sent from my SM-G965U using Tapatalk
     
  10. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    Can you let me know the steps to reproduce it? I am curious if I can figure out what is going on (unless you already know)
     
  11. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    Put YouTube on a mini with DHCP on and make sure there is no reservation set on your router. It seems like if YouTube is running when DHCP releases and renews is when it happens for me. Your lease time may vary from 1 day to multiple or only few hours.

    Once I put a reservation in on my router, it doesn't happen any longer. It doesn't make sense to me, but those are the symptoms and results I have found. A static IP on the TiVo mini may do the same and resolve, but I didn't try since it is annoying managing and entering network info. By reserving it tells the TiVo to never release the IP and so it seems to solve it for me.

    Sent from my SM-G965U using Tapatalk
     
  12. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    Fascinating. I totally believe you, but this makes absolutely no sense to me. Typically a reservation is invisible to the DHCP client. A reservation just makes sure that any time a DHCP request is made, same IP is issued and that that IP is not issued to someone else if the box is offline for a while. If you have a reservation and DHCP lease time of 1 day, it will still request a new IP every 12 hours, it is just that it will always get the same IP. Static IP is different, but reservation should not matter. My gut feeling on this one is that it is a resource issue. YouTube chews up too much resources or somehow interferes with DHCP completion, and it gets into a loop - but that would not be fixed by reserved IP... Like I said, fascinating... I will try to recreate..

    Last 2 questions - did you by any chance get a chance to capture the traffic storm to see what is going on? And what is your DHCP server? By any chance are you using Google Wifi? Thanks.
     
  13. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    I totally get what you are saying and I wouldn't believe it either since what you described is what I would expect. I am using Google Wifi with three pucks connected over Gigabit wired network, not WiFi mesh and multiple (3) gigabit switches throughout the house. When this happens, my network is slammed, all devices wired go offline but WiFi devices continue to work fine. It could be something to do with Google Wifi also, not sure. I did not take a tcpdump yet since I found this as a workable solution fairly quickly. Another interesting thing is only the TiVo minis. The Roamio and Bolt don't exhibit this behavior so I doubt it is a resource issue, but more of a network stack/driver issue with the Mini and hardware and software. I work in tech also, FYI.

    Sent from my SM-G965U using Tapatalk
     
  14. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    Like I said, I totally believe you and you seem to have done your homework, I am just trying to figure out how it is possible and compare notes. What is most interesting is that I have pretty much same network setup with Google Wifi (I wonder if Google Wifi + TiVo Mini combination is why this is happening). On Tivo front I got a Vox-Mini, non-Vox mini(A93), and a Roamio. As far as I can tell, this only happens on non-vox mini - which happens to be connected to a switch right next to main Google puck. Just like in your case, my wired network goes dead and wireless is fine, which kind of makes sense, except if it was a broadcast storm, I would have expected it bridged to wifi - though it may be that Google Wifi is smarter than that. BTW, I also noticed it dead with black screen, but I usually attributed that to a bad HDMI connection, though I always fixed it with a reboot of mini - so now I am guessing it was the same issue. It was less noticeable because most visible things on my network are wireless, which is why I didn't always notice the DOS...

    @Kremlar , any chance you have Google Wifi too - just trying to see if this is a common thread...
     
  15. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    My Mini when this happened also have a black screen and are unresponsive in this state, the light is solid white if I recall, but it's been a while. I had to power cycle the Mini which leads me to believe that it is a TiVo issue and not Google Wifi. Additionally, I would expect Google to resolve any issues quickly with their update mechanism where TiVo is very slow to respond to anything non revenue generating...

    Sent from my SM-G965U using Tapatalk
     
  16. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    One other thing I have noticed. I have a Mini in the garage where I have a TV, I do not use YouTube on it, but I do use Plex and Pandora. It has happened on this device as well when running apps, which matches the issue with YouTube. This leads me to believe that it is something related to when apps are running vs normal DVR functions and it switches modes.

    Sent from my SM-G965U using Tapatalk
     
  17. Hichhiker

    Hichhiker Just me.

    532
    12
    Apr 21, 2002
    I am guessing it is a resource issue. We know these boxes run out of memory/cpu when running apps. Which is why they take forever to load. In comparison, the Vox Box is blazing fast and can load multiple apps because I suspect it has a lot more memory. Maybe if it runs out of memory, it starts sending out corrupt DHCP packets which ends up causing a network storm. But that does not explain why DHCP reservation works... I wonder if Google WiFi does some sort of a ping before issuing a non-reserved IP, to make sure it is not in use, something that triggers this bug? That could be the issue....
     
  18. jaredmwright

    jaredmwright Member

    282
    9
    Sep 5, 2004
    Bay Area
    If you do a packet capture, would be interested to see what you find if anything interesting. Good luck!

    Sent from my SM-G965U using Tapatalk
     
  19. kdmorse

    kdmorse Well-Known Member TCF Club

    7,491
    1,582
    Jan 29, 2001
    Germantown, MD
    Two random notes about bugs that existed in the distant past. Just things to watch out for, in case they are in any way tangentially related to the subject problem.

    A) There was at one point a bug where the Tivo software itself would hang. When it hung, it did so in such a way that the ethernet port was still operating, still accumulating packets, but the kernel was not pulling them out of the receive buffer. The buffer would fill up, and in response someone (it was never confirmed whether the kernel was doing this, or the ethernet adapter firmware itself) - would send a ethernet global-pause frame on every packet it saw. This was a pain to catch, because the switch itself was consuming the PDU's, not propagating them out to other devices on the network. Essentially, it killed the switch it was plugged into, without killing any other switches (as global pause frames are not retransmitted), and without making it to any other nodes on the network to capture. It was finally caught by reproducing the issue while connected to a hub.

    B) There was at one point a bug where the DHCP renewal engine, if it saw a packet it didn't like at renewal time, would go into an infinite loop and just seize up completely. I have no idea what 'real world' conditions would trigger this, but you could trigger it when running your own dhcp server with certain typos in dhcp.conf, especially in the dhcp_server field. Synptoms would be simple, you'd be fine on boot, fine on IP address acquisition, but at lease renewal time you'd hang.

    And B could cause A. (Although it was far from the only source of causes of A at that time).


    *** Memories very fuzzy. If any of these look relevant, I can go dig up the email chains in question...
     
    MikeMacMan likes this.
  20. kdmorse

    kdmorse Well-Known Member TCF Club

    7,491
    1,582
    Jan 29, 2001
    Germantown, MD
    Also, any time DHCP renewal is a suspect.. You can always try to make the problem *worse* for the sake of troubleshooting by cranking the renewal interval down to 5 minutes.
     

Share This Page