Brandon Sanderson’s first law of magic is, as follows: “An author’s ability to solve conflict with magic is DIRECTLY PROPORTIONAL to how well the reader understands said magic.” (he also has a second law and third law online) Sanderson goes on to divide magic in fantasy into “soft” magic — its powers and limits are unclear or undefined — and “hard” magic, where the power of magic is detailed and clear. The point of his rule is that if the nature of magic is vague, using it can easily become a deus ex machina—just pull a spell out of the mage’s butt and presto, crisis solved!
This is, of course, a speific example of a general rule about pulling stuff out of your butt at the last minute. You don’t solve a mystery by having a witness turn up out of nowhere who nobody knew existed. In an SF mystery, as Isaac Asimov once observed, you don’t use SF to cheat (“Little did you know you were sitting in my new invention, the Truth Detector Chair.”).
For a good example, consider Harry Potter and the Prisoner of Azkaban. At the climax, Harry saves the day by driving off the dementors with his patronus. This only works because a)we already know what a patronus is and how it works; b)it’s been established it can drive off the dementors. The only question is whether Harry can master the spell (surprise, yes!). Likewise the Time Turner which plays a vital role at the finish has been established, less obviously, earlier in the book.
Of course Rowling, as Sanderson notes, isn’t exactly writing hard magic. While the broad principles remain consistent (to work magic you point a wand and so forth), the series constantly introduces new spells, talismans, powers. But while they often advance the plot, like the pensieve in Half-Blood Prince, they don’t solve anything. That comes down to Harry’s brains and courage.
I definitely don’t think rules have to be spelled out, if they’re conveyed by the story—in fact, I think magic often works better that way. In Robert E. Howard’s People of the Black Circle (don’t know the cover artist, all rights with current holder) we get one specific explanation—the Black Seer’s point man, Kerim Shah, can’t kill the king of Vendhya at a distance without a great deal of work. Beyond that, it’s all inference: Kerim kills people by hypnosis, a conjured spider, poison gas, magically enhanced strength. It’s obvious that Kerim Shah can’t simply point at Conan and zap him with a magic missile.
For an example of breaking the law, there’s Dorothy and the Wizard of Oz. At the end of the book, with Dorothy and the Wizard trapped, Dorothy suddenly reveals that Ozma checks up on her every day. All Dorothy has to do is make a secret sign and Ozma will teleport them to the Emerald City, and so it happens. Even though Ozma’s powers are fairly clearly defined, it’s still rather deus ex.
And then there’s Lost (I was rewatching it earlier this year for the time-travel book). The series is pure soft magic, as I’ve noted before: things happen because weird impossible things happen on the island and that’s all the explanation we get (the golden light is not an explanation, it’s just more weirdness). If they ever had a hard concept of the island’s powers, I don’t think they stuck with it.