I think taking care of these “issues”/factors is important from the beginning (AKA MVP):
Mobile design – Our site should be seamlessly available from mobile devices. We should start developing for mobile from the beginning (not years later!) and possibly even “mobile first”. I think some few complex pages (e.g. mod/dev tools, especially involving large tables) can be made desktop-only for now.
Accessibility – Our site should be accessible to everyone. This means: semantic html, good choice of colors, supporting keyboard navigation, …
Graceful Degredation – Users who are not using supported browsers (too old, unconventional) should be able to use our site’s basic functions: reading, asking and answering. It doesn’t have to look as nice as in supported browsers, but it should work.
I don’t think this is really a disputed thing, however I am just noting it here, so that we won’t forget it. Have I missed something?
Ideologically, I absolutely want the site to be accessible to everyone, including blind people, people with a “ridiculously” slow Internet connection, people with a “ridiculously” small screen, etc.
Practically, let’s not require perfection in the MVP, otherwise we’ll never be done. Doing this right requires a lot of testing by competent people.
So while some of this translates to concrete requirements (e.g. don’t do fixed-size layouts), it’s primarily a set of general guidelines (e.g. don’t rely on the latest feature that Firefox copied from Chrome last week). We don’t need to validate that every single HTML element is reachable with Tab, but we shouldn’t knowingly use non-tabbable widgets.
As someone with some accessibility needs, I agree with @gilles. If we avoid hard-wiring sizes (page width, font size, etc), that’ll already be a big help. If we make sure that all of our graphical UI elements have alt text, we’ll be even closer. I don’t know much about screenreaders and keyboard navigation, so I can’t comment on first steps there.