The smaller communities, especially, work on SE because they’re made up of people with a common purpose, who get to know each other and who keep bumping into each other. Newcomers are always welcome of course but there’s a common, recognizable core.
This!
You nail down the feeling I had being a high-rep user (top 100) on electrical engineering.SE (we have another one from there @olin, top one last I checked!).
Although it is a very technical site, there was a sense of community among top users (maybe not everyone of them realized that), even when there were some opinion clashes on what constituted on-topic questions, or whatever.
There was consensus built around some issues, and there was discordance on some other. But there was a recognizable and yet intangible border between the core community and the “outsiders” (in a good sense, as “not yet integrated in the community”), some shared “values”, if you want.
Lately SE moves has greatly let me down about how these de facto “communities of values” (top users, real enthusiasts, whatever) were “sacrificed” on the altar of “inclusion”, political correctness and business.
I really do share your view that “big bucket” communities seldom work. That’s what also why I stopped contributing to SO and concentrated on EE.SE. I had extenuating conversations on meta or in comments about, for example, why something was important for a C programmer and still got rebuked from much higher rep users that maybe hadn’t written a C code line in their life (according at least to their profile and activity pattern).
Moreover, another issue with too big communities (or too non-homogeneous ones, at least) is that the feeling of community get lost and gamification, which indeed is an important part of the whole thing, becomes the biggest thing. So you see frustrating FGITW (Fastest Gun In The West) answers skyrocketing with votes, while you are still in the process of writing a good answer that will never make it to the top five because, in the time you finish it, there are already a dozen of highly voted ones (and sometimes one crappiest than the other).
So, yes, I hope you come up with a viable substitute for SE network that allows communities to thrive.