How do I evaluate the answers for a problem, like helpful, nonsense, no use etc.?
I would hire people to do the rating, such as via human powered API or mechanical turk, then the attached machine learning should be able to learn from them like spam filtering.