I noticed this several times: someone posts an answer and gets downvoted pretty quickly without any comments about what is wrong with it. This is pretty disheartening, especially for new contributors who are likely just trying to help and making a honest effort. I understand that the faults may be blindingly obvious to the experts but I think letting the person trying to help know that there’s a mistake is at least polite. If you know the correct answer, by all means post it and let the OP decide which one to accept but silent downvoting (especially on a single answer) doesn’t really help anyone IMO.
I share your frustration, but:
People can be busy and they are offering there services for free.
A silent downvote is still valuable information.
I would still prefer people to downvote silently than not to downvote at all. At least then we know (hopefully) that the answer is suspect and we don't learn bad information.
Most times another answer will come along which is well upvoted so the OP gets the information they require. That doesn't help the person who wrote the faulty answer, but I guess if they still really want to know what's wrong they could post another question. Far from ideal, I know.
Maybe all those extra questions would discourage the silent downvotes, but I doubt it.
In response to Eiríkr Útlendi's comment:
Sadly, my Japanese isn't good enough to distinguish cases where the downvotes are just petty or misinformed.
One solution might be, if the members of the site that we trust (we know who they are) were to comment on the down voted answers saying that they saw no problem with the answer. Just upvoting the answer isn't enough because we don't have enough users to make the statistics relevant, but a simple comment from a trusted user would be enough to restore my faith in the answer. Hopefully that would be less of a burden than having to explain why someone's answer is worth downvoting.