I'm peripherally in the industry, but even I can say middleware is huge. Speedtree, as you mentioned, shows how niche the suites can be. You also have FMV players (especially on mobile) and of course physics engines are huge. Look for Havok and you'll see it powers most AAA games. I don't know specific pain points you can ease, but if you can generalize it, game companies will probably pay for it.
Also, middleware has to be a node on the outside of the graph. It has to substantially all of a problem, not just most of it.
So a fully functional physics engine, a complete procedural tree generator, a self-contained front-end system, a reasonably complete animation package, that's fine, that's great stuff.
Sixty percent of a solution plus an API is not okay.
A complete solution to a problem (no matter how niche) with the smallest integration surface is key.
potatolicious mentioned animation systems http://news.ycombinator.com/item?id=2303453 Is this an area that is just not very important (wont buy) or one that just is low on the list of todo items? speedtree begs the question of what about speedFOOBAR such as buildings, cities, caves, worlds, fish or how about speedanimal (generating animal + animation)? Is the big place to find a nitch in Art or are there better places to invest such as physics engines?
What about audio? Is that an area that is uncharted or an area that is low on the list of items?
If you want to know what sorts of libraries the game industry wants to license, take a look at RAD Game Tools, they are the masters of this stuff. Some of the stuff they are currently focused on are large data-set compression/decompression (Charles Bloom) and I would guess some sort of tool/library to generically handle sparse virtual texture rendering (Sean Barrett) which is similar to id's megatexture technology... though the SVT guess is more speculation on my part than public knowledge.
In any case, this market is probably not a great one to get into unless you are a deep subject matter expert in one of these areas (or you want to become one and have a lot of time to spend doing so), the bar is set pretty high already and few game development studios are willing to put any effort into exploring solutions that aren't either well-proven or from people they already know and trust (the game industry is still rather incestuous).
Spot on, I think; I would add that as well as being a subject matter expert, you will also need to be intimately familiar with the hardware architectures of the Xbox 360 and PS3, since 'optimized' solutions for these platforms will need to be one of your selling points.
