Should we join forces with TopAnswers?

They’re supposed to work that way. I just wanted a sanity check that we can do that here. For example, TopAnswers has four post types that I know of: question, per-site meta question, answer, blog post. Posts have scores (number of stars), but those don’t correspond 1:1 with votes (high-rep users can vote more than once). We might have a different list of post types, and we’re recording votes and assuming one vote per user. Can we talk? I think we can talk but wanted to check.

I don’t know if they have any API yet. But if/when they do, we can talk to it. As an example, they might well store questions, meta questions, answer and meta answers and blog posts all in one giant table but present them differently to the users and also in the API. The underlying structure matters but it doesn’t really matter.

I’ve dealt with a ton of interfaces between systems - from others to my own mostly, but also from mine to others (customers sometimes leave me but I help them go properly) and between other systems. I’ve done that with APIs (obviously that’s the only way we can get data from SE), direct database manipulation, captured “printer” data, web scraping, OCR, etc. Even when the source is not cooperative, there is usually a way to get it done. And when the source is cooperative, piece of cake.

Sorry for not answering to some mentions on the forum over the last days, fellows. I’ll have some time to get up to speed here tomorrow and on the Wednesday.

Hi, I’m Jack Douglas from TopAnswers, and it sounds like it might be useful me posting here. I’m also happy to answer any questions you would like to direct at me.

Some history:

• I was an elected SE moderator on ‘Database Administrators’ from 2011 until October this year. I also served as pro-tem moderator for three years, until 2014, on the niche Biblical Hermeneutics site. As such, like many of you, I know the ropes on SE fairly well (and I often had vague thoughts about what makes the network work so well, and what might be done differently to make it better).

• I had my reasons for resigning as an SE moderator, but my reasons for wanting to create an alternative are not the same — SE the company seem to be heading in the wrong direction on multiple fronts, otherwise I’d be happy to continue to contribute there. Moving is painful. I imagine most people here on Codidact also have various overlapping reasons for coming to the same conclusion. Perhaps none of us has exactly the same reasons — but I think it is likely that many of us share the sense that if the platform is owned and run by a for-profit company, eventually the contributors will be seen by the owners as part of the ‘product’ rather than ‘community’.

• We (my son James and I) started developing TopAnswers on 14th October. We had a particular focus in the early days of getting chat right, not because that was the eventual goal but because we saw it as “essential to getting core people here”. We are now firmly focussed on developing the Q&A side, and always will be from now on — though we have no intention of neglecting chat the way SE have done.

• I first became aware of Codidact on 11th November. But only had a quick glance at the time.

• By last week we had started to think where to start regarding other communities (beyond ‘databases’ and ‘meta’) to plant on TopAnswers. I had been toying with the idea of something like ‘*nix’ and began to speak to a few people I know about the idea. While looking for a way to contact someone outside of SE, I wondered if they might be active on this forum, dropped by, and found my way from here to your Discord channel. Up until that moment I had no idea how many people you had involved, or how developed things were here.

• So I finally did some research and decided that, while it probably seemed too late to merge the projects, it might be nice to share ideas, and perhaps more. Not sure how to permalink to Discord chat, so I’ll quote my first message here:

  I’ve only recently become aware of Codidact, and it’s perhaps already too far down the line for the two groups to work together, but I can see scope for cross-pollination (of ideas at least) with the site we are developing. There seem to be a lot of shared aims, and some similar ideas about execution. If it would be appropriate for me to post a link in here, let me know and I will do so. We are mostly from the dba.se community, but are planning further communities, possibly starting with ‘*nix’.

• Some friendly chat ensued, and some very valuable feedback on our UI was gratefully received. One piece of feedback I’d like to respond to here, as it’s really important, and is, I think, a misunderstanding (see below).

• Monica has had a good look around our site, and posted this post, and here we are!

I’d like to say a word about my hopes. I hope there will be at least one successful, thriving, place for SE-like focused Q&A communities, which has the kind of ethos that Codidact and TopAnswers share (it does seem that there is a very large degree of overlap in ethos — open, not-for-profit, real communities etc — the differences are practically all about implementation, which is important too of course).

I will be happy if that turns out to be Codidact, or TopAnswers, or both. I will be happy in a very different way in each case of course! But my top goal is that no-one spends another decade investing their kind-hearted effort into ‘building a library of knowledge for the internet’ and then finds the ground shifting under their feet and their contributions being monitized. Or at least that no-one is in that boat because they had nowhere else to go.

I’m not going to try to defend all the design decisions we have made at TopAnswers, although we value feedback and nothing is set in stone. I would like to list a few guiding principles we are operating on though, as it might help see where we align:

1. Clean, focused Q&A is the aim. Answers sorted by some sort of voting mechanism is vital. SE got that so right. I wouldn’t put it past them to ruin even that in the future somehow, who knows. Maybe they will start awarding rep for comment upvotes :S
2. We are trying not to blindly follow anything else SE have done. Familiarity can be really helpful but cloning their every feature will mean bringing many of their problems. Example: we are not convinced that downvoting is necessary, even though I see the value it does have. Even if something is useful we might not implement it if it also adds a significant degree of complexity or drama. Or we might implement it later when we are proved wrong, but in the meantime we won’t rush in.
3. KISS. Why have a whole dedicated separate per-site meta community when we can just label posts as ‘meta’ and treat them more or less the same as any other post? Why bother with a separate blog when we can label posts as ‘blog’ and keep them ‘in universe’? We know why SE write blog posts rather than posting on their own platform but we actually want honest engagement.

Finally, as I mentioned earlier, I want to respond to one piece of feedback with a clarification. It is the idea that we have promoted chat on TopAnswers, at the expense of Q&A. It’s true that we are doing things very differently from SE regarding chat. Chat and comments are merged and very integrated into the experience. However this is not a promotion — the opposite is true. SE comments interrupt the flow of clean focused Q&A and that works against the key factor that makes SE good. We’ve taken all comments out of the Q&A flow. We want to make chat better, and community more integrated with Q&A, but we will never downgrade Q&A.

To make a reference to the movie Life of Brian, if we want to overthrow the Roman Empire, we can’t have both the People’s Front of Judea and The Judean People’s Front. They’ll end up spreading resources thin and maybe even fight each other instead of the Romans.

How many crappy Q&A sites are there already? We don’t need two more. We just need a single good one. To succeed is easier said than done, of course.

@Lundin, quick question if I may, would you like to see SO broken down by language (so ‘C’ and ‘C++’ as distinct communities — but possibly with some deliberate cross-linking like an opt-in mechanism to see new questions from other, related. communities on the various homepages) if/when the Romans are overthrown?

Thanks for taking the time to talk with us! How do you feel about sharing data / open API between the two platforms (Codidact and TopAnswers)?

How do you feel about sharing data / open API between the two platforms (Codidact and TopAnswers)?

I think we should do this. Someone mentioned syncing, which I don’t want to do (at least not at our end), but not because of any objection in principle, just because it is always way harder than people imagine it will be.

We are importing selected content from SE and it would be a lot easier if there was a more open API we could use. At the end of the day we want to make sharing content as easy as possible not as hard as possible, because that is the spirit of the licences that contributors have shared their content with us.

At the very least we’d be happy to provide a private API for Codidact to pull content over.

Thank you for posting here!

I (and I think we) share those hopes too. In addition to what you said about not pulling the rug out from under the communities that build the content by turning them into the product, there’s another community concern: I’d like communities to be able to gather in one place instead of scattering to the four corners of the earth. We, the members of the communities, are stronger together. We cross-pollinate, we learn from each other, we help each other – if we share a platform. But if some communities are on TopAnswers and some are on Codidact and some are on QPixel and some stood up their own sites using Question2Answer and some did ten other things, we lose that.

One size does not fit all. I get that. SE didn’t work well for some sites like Code Golf, from what I’ve heard. But if we build a platform that’s flexible enough to welcome almost everybody, that allows the per-site customizations that sites actually need, and that still works as a cohesive whole, we can support not only each community but a community of communities. That’s my dream – not just to rescue Writing and Code Golf and DBA and whomever else, but to help us stay together. We’re stronger together.

Agreed. We can learn a lot from SE, both what to do and what not to do. We have many of the same aims and will end up with the same principles in a lot of places, but this isn’t a clone.

I really like that. I thought it was neat when I visited your site and the blog posts were right there, in context.

We’ve been talking here about comments and chat too, and the idea that chat shouldn’t be off in some other place but comments/chat can’t take over the question pages either. We’ve talked about in-page affordances, whether that’s tabs or collapse/expand or something else. Your idea of putting it in a sidebar is, to me, “something else” and works toward the same goal. I have some implementation quibbles (as I’ve posted on your meta), but the approach is sound, and one of the reasons I think we aren’t so different even if, currently, the first impression of your site is “chat-dominant”.

Finally, thank you for agreeing to an API between us! If it doesn’t make sense to collapse onto one project, let’s at least keep those content-sharing paths open!

Thanks for the write-up!

Everything I’m reading here looks compatible with what we’re planning for Codidact.

Do you have a list of these decisions somewhere? https://topanswers.xyz/meta doesn’t have a lot of stuff, but presumably you were talking elsewhere before.

Same here. We’ve been debating the details, but nobody’s proposed an alternative to voting (although we’re talking about mechanisms to complement and modify voting).

Neither are we. Most of this forum is about “should we do this thing like SE or different?”.

We’re divided as to how downvoting should work, but the idea of doing without downvotes hasn’t gained any traction here. And I do think it’s necessary for an MVP: how else are you going to indicate that something is wrong?

Why bother with labeling a post as blog? Just make it a self-answer.

We’ve been going a similar route. I like what I’ve seen so far on TopAnswers, but I wonder how it’ll scale with thousands or millions of users.

Here on Codidact we’ve been talking so far. We haven’t started coding. Do we have any incompatibilities? If not, why would Codidact start coding, rather than join the coding effort on TopAnswers?

The major incompatibility is tech stack. TopAnswers is PHP; the stack we’ve picked here is .NET Core. Going by a read of the #skillset channel, we have plenty of people familiar with C#, but not very many familiar with PHP.

There may be other smaller incompatibilities too - which may be reconcilable or avoidable, or may not. However, the tech stack disparity significantly changes who is able to contribute, and from what I can tell will result in us having less effort available to put into TopAnswers than we have available to put into Codidact.

Ultimately I think it will have to be down to each contributor. As organisations, we’re in agreement that we’ll make sure we’re compatible somehow - possibly a content-sharing API. As contributors, we each need to pick what we’re most comfortable contributing to - if we have folks who are happier in PHP than in C#, TopAnswers may be the choice for them; likewise if folks are more interested in the way TopAnswers is doing things. If folks are easier with C#, or want to build something where we can choose how we do things as we go, Codidact may be their choice.

Most people have skills in more than one area. So while the set of people who participated in that vote here favored .NET Core, that doesn’t mean they wouldn’t be able and willing to contribute in other areas too. We’re going to need code (and technical design), both back-end and front-end, and database design (which I believe is independent of the implementation language), and lots and lots of functional design yet, including what’s configurable and what effects those configuration decisions have.

What are our invariants, the things we will not budge on? I mean at both a product level (the thing we build must do X) and at a people level (I will absolutely not work on a project that does Y). Let’s identify them and then see how they align with TopAnswers’s invariants.

Maybe we can’t all align, but let’s not assume we can’t without examining the premises. How many people who are here now and ready to contribute were even here when the tech-stack conversation happened? How many of those who weighed in did so believing that the choices were otherwise even (no external constraints)? “Choose X or Y where we’re starting from scratch” is a different question than “choose X or Y, and X means we join forces”.

I realize that, as someone who will not be writing code, I might come across as saying that people’s tech preferences for a volunteer project should be set aside. I’m not saying that, and I do not want anybody to feel that I’m dismissing tech concerns. I"m saying that the landscape has changed and can we please re-evaluate?

The initial vote was based on a very small group of people, and I suspect skewed a little by knowing that SE uses C#/ASP.NET. Personally, I know no C#/ASP.NET but quite a bit of PHP. At this point our group has grown, but new people joining know that we are planning on C#/ASP.NET so they self-select based on that or (if they want to code but not C#) they move to front-end dev.

In any case, I think there are more differences than just the tech stack. But I do think we can work something out, whether it is API or possibly more.

Sadly not — some of it is starting to come together on meta but other things were discussed somewhere in chat or in the early days just existed in my head.

Blog posts are slightly different on TopAnswers in that you can’t ‘answer’ them. We have a half formed plan to allow questions on blog posts (with their own answers) — but we’ll only do that if it doesn’t add too much complexity, so we’ll weigh that up as a community at some point.

We are wondering that too. We are trying to think ahead — my view is that it is not the language choice that will ultimately decide how well we scale (ie C# v PHP), but how the database is designed. We can scale out for reads but we need to be careful about things that change state on the database unnecessarily. There is no CRUD access to the database from the application layer — everything goes through a carefully constructed transactional API (views for reads and functions for state changes). I think this will make everything easier to control if/when we start to scale, but of course it is all untested yet.

I am about to import 175k chat message from a particular SE room which will be some sort of test of how transcript searching scales!

Not quite sure what the benefits are. There are some advantages with SO model: every user can moderate the posts without domain expertise, and not having distinct communities per language prevents sub-cultures from forming. If sub-cultures cause different rules regarding posting, formatting and other things that are not related to the specific technology of that community, then that is harmful.

However, what SO is missing is a uniform FAQ system per programming language. And so every language community on SO solve this in their own way (again, sub-cultures). C++ has a meta tag for FAQ, C uses tag wiki, some other languages use external sites and so on.

As a hobbyist website-builder:

• I am gravitating much more towards the typical free/open projects like postNuke and other open source webpage management systems,
• I am more capable in PHP. I have heard of C# but not more than that. I have little idea how it relates to C++ in which I did some stuff (also hobby, and a bit for University introductionary courses).

This describes me, but I believe it can be very much generalized to any other (low level) website amateur. I consider C# much more as a platform for relatively more professional websites and is gonna be an entirely different public.*

Example:

If you are a professor or some other worker in academy/high-school and wish to make some Q&A then a Codidact instance will be much too specialized/complicated to host on your own and you will not be able to do it without help from (expensive) website administrators (if the university has any). So, you will make something in a simpler language like python (see for instance the biostars Q&A).

Now, I understand that casting the Q&A format into something like what phpBB did for forums is not very appealing. Or at least, it would generate a wild scattered multitude of many different (but also disconnected or only loosely connected) Q&A websites (like you have many different blogs, forums, etc. all over the place).

This does not seem to be the mission of most people that have an interest in Codidact. This mission is, I believe, much more like creating a larger platform for Q&A that is able to be on par with SO/SE. (although SO/SE seems to go in the opposite direction towards these smaller sites with their ‘teams’ product; so we may learn from that and try to understand that motivation).

So, I believe that there is value for both different systems and both different scales.

A lot (more and more) of the online world is happening on the big (dominant/monopolistic) platforms, and it is important to have/copy something strong impactful like that**. Yet, while SE/SO likes to boast about how important they are, we should not treat lightly the value of the many little small forums, discussion-boards, blogs, etc. where you find the information as well (I guess that at least half the work by SO/SE is just their SEO and not so much the content itself).

On stats.SE we have a specific page that refers to all these little sources and all taken together they might be considered just as rich (or better) source of information as the 150k Q&A on stats.SE. (My personal experiences: for questions about R coding or Python I much more often end up somewhere else then StackOverflow, and when I was still working with PHP then the forums about PHP and the PHP site - I do not know what exactly anymore, but it was a very clear documentation/information site - mostly helped me. For questions about Ubuntu, which was my first experience with Stack Exchange, I used to mostly get information from the Ubuntu forum but somehow google - my default Q&A machine - is directing me to AskUbuntu more often now, but I can’t say that I like it much better there. And still the answers on AskUbuntu mostly directly me to pages outside the SE-network)

If the two systems (large capacity professional - small amateur capacity) are able to generate a strong integration by using similar standards and connections then this would be great and can only make each stronger instead of competing with each other.

*Sidenote: Now, my knowledge is very rusty. If it is nowadays as easy to get a server with C# and a Codidact site running on it, as it used to be ten years ago to get a PHP based content management system running, then this breaks down all the premises on which I based my story. (at least it still pictures the sentiment that might be existing with some, especially amateurs, when they hear C#)

**Also because those bigger platforms are very stable and smaller sites often disappear along with their content. Although this is also true for big platforms, like MSN groups, that are shattered into smaller communities.

My personal experience with that: I used to be frequent visitor of this cocktail forum (https://www.webtender.com/iforum/) which became less popular and a large part of the community was discussing at an MSN groups page but that got shut down. Now I am not so sure where people do anymore (but I am also less interested in being a frequent visitor of such pages) and it is possibly some google groups page or conversations scattered over Facebook.

The bottom line. The internet seemed to have developed by several violent waves destroying the little piles of sand on the beach, and now we are mostly left with a few big piles like Facebook/twitter/google/instagram/etc. That’s the competition we are facing so we need to have some framework that is able to build such pile of sand as well. But, if we keep a strong connection with TopAnswers then we might be able to better serve the small communities and not become this big pile of sand with very large and empty holes next to it.

Hi. The first comments by Manasseh and Art do a good job summing up my personal view on this proposal. On top of that, I agree with every other message here, to varying degrees of course. That being the case, my comments will be largely limited to the technical aspects.

Let me start by congratulating Jack Douglas and the team for the effort they put into it. They have something up and running, even if at alpha state^(?), made from scratch (I presume) and this deserves appreciation.

While our goals certainly align - and, barring certain implementation specifics, they align a lot, I believe the degree to which we can join forces is indeed a bit limited.
In other words, I don’t see a merger happening, not anytime soon, at least. But other than that, collaboration can be varied: their technical implementation requires, for a large part, a diverse skillset than we do; and since we might have contributors from our side who happen to be much more experience with these, while not so much with our own choice of technologies, some of which might be willing to actually lend a hand in the TopAnswers project - at the same time as building the necessary familiarity with our own tech stack so as to be able to make more significant contributions -, I would certainly look kindly upon those of us who decide to collaborate to their project as much as they can. This is in addition to the ideas that were already mentioned.

It also happens to be the case that TopAnswers needs some UI work done right now, as part of something called the frontend, while we just cannot work on this front ourselves just yet. We don’t have a structure - we’re very close to having one, but it isn’t here yet. When it is, it’ll take some time (for those who’d like to have a rough idea of how much time: let’s say, about 1.5 months) until some relevant UI implementation can take place. What we do need as of now (as well as in the long term of course) is design/layout, but then again it’s yet another, different skillset - albeit there is overlap, but the truth is most front end developers are not too happy, or even capable (myself included) of doing design. Considering that, there it is, another possible path for cross-collaboration at the software level.

From our side, we’ll be needing contributors with experience in database modelling, preferably with Postgres in particular, as well as software engineering/architecture, to get the base project off the ground. We’ll be needing this now. It is my understanding that TopAnswers is already past that point, I don’t know what the size of their team looks like and whether some of them would be willing to contribute with our initiative too, but would very much appreciate if this was possible.

In terms of our choosen technology stack compared to theirs, I maintain that ours is the one I believe to be the best fit for getting to where we all want to be in the future. I am a reasonably competent PHP developer myself, and just a couple of months ago, had a bit of a struggle in deciding between it and C# for one of my next projects, as can be seen from this post on Reddit. That led me to evaluate a lot of factors that makes large scale web applications successfull and compare some of the most popular examples (and some not popular at all), their codebases and implementation choices.
The divergence goes quite deeper than just PHP actually. I see little point in enumerating every difference here at this moment, but the main one is the presentation/UI mechanism, that runs on top of the PHP backend and is mainly powered by JavaScript. Based on my experience, I, personally, cannot endorse this approach and find it unlikely to potentially attracting enough interest from the community at large, as well as keeping them happy in the long term by providing a consistent, scalable and cohesive experience.

@JackDouglas, if I may ask, did your team consider using an existing implementation prior to starting your own?
If I were to go the PHP route, Question2Answer would certainly be my pick. It looks mature enough and, for running something on the short term while developing our own solution, it should require little modification. For using as a basis for a long-term project, however, I would certainly want to change their database engine (it uses PDO which does ease things a little bit), as well as make fine adjustments to its structure, but still.
How did you come to the decision of writing your own, and what were the leading factors to opt for the specific set of technologies you have chosen?

Thanks Marc, that’s kind of you to say so.

Please draw my attention to any related posts here. Apart from that if you’d like the current database schema and internal api layer please say so — it might possibly be useful just to trigger some ideas about things you may not have considered yet. I’d love to get you sold on the internal api approach too! It’s a database<–>application api, and with enough discipline, that approach has some neat benefits (eg atomic replacement of the entire api layer for releases, more complex rule enforcement complementing DRI, etc).

It’s nice building on a ready-made foundation, but I can’t help feeling that if we’d gone down that route, I’d still be the only person using TopAnswers. I guess I mean we are going for a ‘people-first’ approach to building the site rather than ‘technology-first’ — we started by building a chat room for some of the top SQL Server experts around, who were looking for a new home. We are now building a databases site with them and others from dba.se who we know well. We are running a private beta for a TeX community — at each stage we are trying to build something that they (and perhaps only they at this stage) can get excited about. There are loads of rough edges on TopAnswers — we should label the profile page with “this is not really how the profile page will look in the future” for example — but the core Q&A and chat integration is getting smoother and the people using them are getting happier!

You can never please everyone (of course), but the aim was to make an ever-widening circle of people very happy. I didn’t see any point in building something that a lot of people might think was “ok” — I’d rather have fewer people to start with, but passionate about it. It’s very hard to build that kind of vision on top of a large foundation that is solid, but kind of ‘pedestrian’.

TopAnswers divides opinion, but not (I would argue) because it is primitive, or wrong, but because it is undoubtedly different. More than a few times I’ve seen comments like “I wasn’t sure about X at first, but it grew on me really fast”.

I’m a database guy, as you know, and as such I tend to regard UI, and indeed applications in general, as disposable fluff. I exaggerate, but you see where I’m coming from . Postgres has it’s limitations compared to commercial databases, but it’s not a million miles behind, and in a lot of ways it is a joy to work with (speaking as an Oracle DBA). One small example: the question search feature on our front page is not quite Google, but it’s very much ‘good enough’ imo — and it’s a doddle to implement with Postgres’s trigram matching and GIN indexes. I’m expecting it to scale well and there are some useful levers to tweak when it comes to the trade off between search accuracy and speed. Postgres ticks so many other boxes of course, being free, open-source, standards-compliant, having mature replication features, etc.

On the other hand, we chose PHP because we know it. I would be equally happy writing the UI generating code (which is all the PHP really does — as much ‘business’ logic as possible is pushed down into the database api level, as Postgres functions) in python/.net/whatever if I knew them as well as I know PHP (which is ‘averagely’ well). I don’t think it would make any difference and I certainly don’t think it’ll affect the issue of whether we’ll be able to scale very much. It would be relatively simple just writing another UI on top of the TopAnswers database — it’s designed that way. I don’t want to put PHP down though, it’s often the best tool for a job, and quite underrated sometimes I think.

In short, Postgres+PHP because Postgres is great, and because PHP works well enough.

@JackDouglas Thank you for all that additional information and insights. I am glad to hear that you are working multiple communities - that puts TopAnswers a big step ahead of where I thought it was.

I am very interested in seeing that. I have a bunch of ideas what I think would work well, and before I get too far in it would be helpful to see how someone else has done it. I’ve seen a lot of talk (by others in the Codidact group) regarding SEDE as a comparison, but it is very clear that SEDE only represents part of the schema actually used by SO.

You can grab it from https://topanswers.xyz/topanswers.tgz — please let me know when you have it and I’ll take it down again. Please feel free to share with others at Codidact, but not more widely (if possible) as we have not yet had a proper security audit.

Probably needs some explanation, so feel free to ask — but ‘shared.sql’ chains the main ‘get.sql’ and ‘post.sql’ api scripts. db.sql is the permanent database objects (and doesn’t actually get run — we just update it when we make changes to the db so it might possibly have some minor errors)