This post by Jonas Kristoffer Lindell presents a parsimonious view of common statistical tests which are, on their own, confusingly and inconsistently named and, taken together, a mess.
Jonas argues that there is a common theme among these tests and that it is simple. Simple to explain and simple to understand.
I highly recommend you give it a read.