Response letters are hard to write, but there's no word limit.
The most difficult to respond to are requests to make the work described more ambitious -- "this extra thing would make the results/method applicable to A, the case I care about, you just have to XYZ" where XYZ requires months or additional samples or whatever. Usually adding a senetence to emphasize that A is not included addresses the objection, but a type of exchange that's oftenfrustrating.
Hardest to write are "not even wrong" or "this is a lab notebook dump, where's the focus?" reviews.
Identifying substantial revisions that would lkead to a better paper is not the reviewers job really, hard to deal with MS that calls for that or reviewers that have that inclination.
Scope of test cases is a tough one.
"include this test case that's out of scope and makes the work look bad" when the test case is a domain that even the deranged would not venture into.
Your testing could be circular or pointless (ie testing may or may not have the right properties to actually test the method) , is it nitpicking or substantive? Could be either depending, and is a very sensitive point for many machine learning papers, where testing practices are being developed by people who think about their algorithms much more than data properties.
The most difficult to respond to are basically "why didn't you do a different study?"
Or "one of your references is a link pointing to an image called 'goatse'".
"Yours is an innovative suggestion."
I think even a dumb reviewer would figure out eventually that they are being mocked.
I'm not actively researching any more, so I only do reviews these days if I'm uniquely qualified to evaluate the article (still happens every once in a while) or it's a book manuscript and I'm friends with the editor/super excited about the topic/want that sweet sweet $300 worth of free books from the publisher (i.e., maybe 3-4 books).
The advice is good on the whole. I'm a little skeptical of point 1, but maybe it's fine. 3 seems like a good idea.
What I'm working on recently is keeping my report in a text file until I get the response, because it's often weirdly difficult to see my initial report in the computer system and authors don't always include enough information about what they're responding to for me to work it out. On the other hand, I don't really believe in giving substantial second reports, that's just way too much work and often veers into the referee trying to write the paper themself. It's the author's paper not the referee's.
10: Move the fuck over eclectic web magazine, a central electronic diary has come to town.