I have never found this to be anything but comical. While I agree with the baseline sentiment that one should, if at all possible, mind one's own business and keep one's yap well and truly shut most of the time and with greater vigilance when it comes to other people's activities (especially when oneself is not involved), and further recognize that my restatement here lacks the pithiness and memorable rhyme scheme of the original, it still rings hollow to me when I hear people - either in media or real life - say it.
Actually threatening people don't have the need, time, or inclination to say shit like this. An actual threatening person gets in your face and says "Stay quiet about this or I'll fuck you up," and you remember the hell out of it while you clean the urine smell from your pants later on. Threatening people don't spend a lot of time making up cutesy rhymes, and if they do, they become less threatening with each one. How long was Ice Cube actually scary, for instance? Pretty much when the Jeri curl went away, so did the threateningness. Now he makes beer commercials. Finally, an actual threatening person doesn't care to make up little poems to scare you, as his time is pretty filled with knuckle-cracking, drinking, and reading monosyllabic hate tracts. So it's not realistic, is my point, and invalidates a fictional character in the saying, and makes a real person laughable.
Even if they've got a shiv. You could wave a shiv around and say this and still lose me. Now it's your turn; in the comments, leave examples of the most threatening things you've heard that actually work.