It’s not about “fairness”, but about votes being proportionate. If 100 users view a post and 20 of them decide that it’s worth an upvote and 10 that it’s worth a downvote, and in both groups there are 5 users designated as “experts” by the software, the post will have a score of +5/-10 (negative score), although in fact it should have +10/-20 or any other positive or neutral score (because it is quite contentious). Your proposal tends to favour downvotes and not a neutral score for contentious or low-visibility posts, which is IMO not good for a Q&A system.
True. Hence I don’t oppose/favour having some limit before being able to cast upvotes (as on SE, where 15 rep is needed) to prevent abuse and people without any on-site activity, but it shouldn’t be conflated with “expertize” or “expert status”, which should be a much higher privilege, if any (=if we decide to have something like [tag] topic expert status, which I favour).
I don’t think that’s a good idea for a voting system. Votes have the problem of being not qualified to express problems with post. There’s a difference between saying “I don’t like this option” and “This option is wrong”. This can only be done via some kind of “vote comment”, which I think is planned (after MVP) in combination with the voluntary disclosure idea.