So, it seems that the need/desire for some cable companies to implement SDV is a result of several possible market/regulatory failures:
Nope, not even a little bit.
0. With some/many cable companies elimenating analog altogether, is there really a need for SDV????
Is there a need for TV at all? Television is most certainly a luxury, and could be eliminated altogether without a fundamental failure of any critical services. The demand for TV services is growing rapidly, however.
1. Bundling of channels into packages.
That has nothing to do with SDV. What's more, regularly scheduled programming only represents a very small fraction of the total bandwidth of a fully developed SDV system. Video On Demand, IPPV, interactive services, and trick-play features such as "Start Over" make up the bulk of the bandwidth on an aggresive SDV deployment. It also means the cost of deploying any individual stream plummets. Already it is becoming practical for a large company to lease their own SDV channel. In the near future, it will be practical for small companies and even clubs to lease their own channel. Your local bowling league, rod & gun club, SCUBA club, or gardening club will easily be able to afford their own channel. The remainder of your point is completely specious.
2. Deterrent by the cable companies for consumers to use their own hardware and not lease cable company provided devices.
That is a much more complex situation - admittedly much of it the CATV companies' fault - than you address here. The impetus for the CATV companies to deploy SDV has a vanishinly small component related to any additinal recevues from leasing their devices, especialy since most MSOs make almost nothing directly from their leasing of STBs and DVRs. As to issues relating to 3rd party devices' inability to handle SDV well, that issue rests entirely with the 3rd party manufacturers and the FCC. It was the CE manufacturers that demanded the FCC force CableLabs to produce and support a UDCP specification. It was the CE manufacturers that did not want to be forced to support 2-way and interactive protocols, and the FCC that caved in to them. I'm not saying the CATV companies are blameless in all this - far from it - but the fact subscriber owned equipment doesn't work well with SDV is entirely the fault of the CE manufacturers and the FCC.
3. Unwillingness of cable operators to invest in and maintain their networks.
You haven't even the slightest clue. MSOs have invested nearly $20 Billion in upgrading their systems, including deploying SDV. Also, you evidently have no idea what goes into upgrading a CATV system. For a comparatively modest investment in SDV, the CATV system can create a system literally capable of delivering an infinite number of channels, with unlimited growth potential for very modest additional outlays in cash. That, or for ten to fifty times as much money, they can increase their number of channels by 125, with zero growth potential.
I'm waiting for you to come up with a single valid one.
Alternatively, is SDV the savings grace for the cable companies that will provided an almost unlimited selection of channels?
Not "almost". In a properly engineered SDV system, there is no upper bound to the number of "channels" that can be deployed.
Isnt SDV basically identical to services like netflix/amazon except it is served over a network only accessible to the cable companies?
No, it's much closer to being very much like RoadRunner, AT&T, or Verizon Internet service. It's a transport mechanism, not a content provider.
What happens to a channel which becomes too popular on SDV and starts using too much bandwidth?
There is no such thing. First of all, if one channel gains in market share, then without fail some other channels will lose the same amount of market share. If a really large shift in popularity occurs, then hypothetically that channel might become better served by a linear QAM, and the CATV company can decide to move it to one such QAM, or not. The most popular Big Band digital modulator serves 8 QAMs, which can represent 16 HD channels and 2 SD channels. Six such modulators can serve the 96 most popular HD channels and the 48 most popular SD channels, or perhaps the 90 most popular HD channels and 81 most popular SD channels. That uses up 360 MHz. Let's assume the CTV system is delivering 90 HD channels on that spectrum, and for simplicity's sake, let's forget about SD for the moment. Now, perhaps you haven't noticed, but more than 80% of the viewing bandwidth is serviced by a mere 10 channels, leavng 80 channels to service no more than 20% of the viewing palate, with the remainder being outside the linear coverage. For argument's sake, however, let us assume over time the national networks no longer have a headlock on the viewing public, however, and the viewing coverage among the top 90 channels is nearly even. This is the worst-case scenario in terms of your proposed situation. In this case, no channel outside those 90 can possibly have a penetration of greater than 1.1%. The actual number is much lower. With a penetration of less than 1.1%, there is a fair chance any given node may not need to broadcast that stream, allowing the bandwidth to be used for some other stream. If some channel other than those 90 gains enough popularity to require more than 1.1% of the bandwidth, then it must be true that at least one of the original 90 now has a popularity of less than 1.1%, and can be moved to SDV, while the newly popular channel moves over to the linear system. 'No big deal, really.
Does this channel then get moved to a regular linear always on channel?
That's up to the CATV provider. Since they purchase modulators that host significant numbers of streams at a time, micro-managing the lineup is probably more trouble than it is worth for them. To the user, however, it is largely transpartent.
For example, if there are 500,000 subscribers watching a SDV channel at the same time, would it not be more efficient to move the transport stream to non-SDV? I mean 500,000 x 12 mbps bandwidth versus 12 mbps to everyone in the regular mux? Maybe I misunderstand SDV?
First of all, it depends on how many subs (actually, tuners, not subs. Most subs have more than one tuner) there are, total. What's more, it doesn't even depend on that. SDV relies upon its gains for a significant number of nodes to not have any users watchihng a channel for a significant period during the day. Here in San Antonio, for example, there are many neighborhoods on the East side of town that have predominantly Black residents, while on the West side there are large areas that are almost all Latin. Both BET and Galavision are fairly popular channels here, but there is a very good chance that a large number of nodes on the East side will not have even a single viewer watching Galavision while on the West side, there will be many nodes without a single viewer watching BET. On the North side of the city, there are probably a fair number of nodes that at the very same moment have neither channel on them.
The bottom line: Here in San Antonio, there are some 300 SDV channels. Of that number, there are only about 60 from which I ever record at all, and only about 30 from which I record regularly. That may sound terribly limited, but then consider the non-SDV channels from which I record - all seven of them - and that in all but 2 cases much less than any of the 30 SDV channels from which I regularly record.