I'm going to go out on a limb here, but joining a gaming group is the most serious commitment you can make in your life. Ok, wait, let me try again. I swear I can say that with a straight face...or maybe not. However, whether or not it is the most serious commitment, joining a gaming group can be a serious commitment all things considered. Sure, you're just hanging out with some friends, rolling some dice, and killing some monsters. But, you're also interacting with people, and bonds and attachments can form. This all leaves it kind of awkward when you've decided that the magic is gone, and it is time for you to go your separate ways.
Recognizing It's Time
When it happens, you'll probably know it has happened, but a lot of us fight the feeling. We look at the game, and we know we're not having fun. We know that we are not getting the same enjoyment out of things. Ultimately, we know that the game isn't how we want to spend our evenings anymore. Now, this can be for a number of reasons. Maybe the campaign isn't gelling for you, maybe you don't like your character, or maybe you have an issue with another player - or even the GM. Whatever the reason is, the point here is to be honest with yourself. If you are not having fun in a game, than something has to change. Sometimes that thing is that you need to go, and there is absolutely nothing wrong with that.
Now, before you go off and drop out of a game that you think you still could like, it might be worth taking some preventative action.
If your problem is your character, than talk to the GM about the character. See if maybe you can come up with a way to make things a bit more interesting for your character, or just to swap it out for someone new. Generally, I recommend against swapping out characters, but sometimes it is the only way.
If your problem is some aspect of the game, then once again you should talk to the GM about it. Try to be polite, the GM probably worked hard on what is going on, but at the same time you want to bring up your concerns and see if you can work with them on it. Don't be afraid to tell them that you're not having fun right now, or whatever your concerns are. Odds are they want you to have fun, and will be willing to work on that.
If your problem is with a player, then it may be worth pulling that player aside and talking to them. Let them know where you stand, find out where they stand, and see if you guys can work things out. I can't tell you how many times issues that seemed absolutely huge to one player, were actually very easily fixed with a 5 minute conversation where no one else could hear. Again, you want to be polite, but you also need to address the issue.
Let me let you in on a little secret about human interaction. Subtly, no matter how much we say otherwise, flat out doesn't work. Oh, and Ladies, this isn't just a guy thing. Take a moment and think about it, how many times has it been obvious to you and your friends that Susie had a crush on Jason - or vice versa - and neither of them had any idea at all? Fact is, when we're in the middle of things we tend not to see the curve balls and sliders coming at us. This is also an issue when trying to resolve problems, and the only way around it is to be direct.
For example, and this isn't even all that subtle. Let's say that Player 2 doesn't like the fact that Player 1's character is constantly picking on them. So, in the interest of preventative action, Player 2 asks Player 1 if they can talk, and says that he doesn't particularly enjoy Player 1 PC's abrasiveness, and it is causing issues for him. Player 1 says he'll try to keep it in check, and the problem is solved, right? Not necessarily.
It is possible that Player 1 doesn't see the name calling as abrasive, but instead as good fun. So, the issue hasn't been solved at all, and player 2 is now more upset because he talked to player 1 and nothing happened.
If, on the other hand, Player 2 directly said "I don't like all the names your character calls my character", then we have directly broached the topic, and a real solution might be able to be found. It might be hard to do, and a lot of people try to hedge or couch their words to spare the other person's feelings, but sometimes you need to be direct.
In the interest of being direct, you may be tempted to give an ultimatum. Something like "If you don't stop calling me names, I'll leave the game." Sure, it is direct, but it also takes on a very hostile tone right from the start. Because of that, you probably don't want to start with an ultimatum, and when you do give one, you want to try and not phrase it as a "I get my way, or else" type statement. Yep, that can be hard. Hell, just a couple nights ago I told a friend online to effectively give an ultimatum (whoops!), but they can be very dangerous things to give.
I'm not 100% sure how to properly say what I am feeling here, so just be careful with your ultimatums. It might be better to couch this part, while still being direct. Something more like "The constant verbal attacks are making it impossible for me to have fun in the game, and there's no point sticking with a game that isn't fun." It is, ultimately, the exact same thing, but the terms are a bit different, and the reason for the second part of the ultimatum is clear.
Make a Clean Break
Sometimes, when you try to leave the group, people will try and resist. Friends in the game will beg you to stay or come back, other people will say that if you go they go - putting you in the position of killing the whole game -, and a bunch of other things can play out. This can make it harder to leave, but it can also give a second chance on getting things fixed so you can have fun.
However, you should try to make a clean break when you can. If you go back to the game, and continue to not have fun, you are just hurting yourself more. This is your spare time after all, and you should be able to enjoy it. So, if people ask you back, then ask them what has changed. Be clear about the reasons you left, and see if something can be done about it. Simply put, if you aren't enjoying your time playing a game, than there is no reason to continue playing the game. That isn't necessarily anyone's fault, but it is something you need to keep in mind.
Either way you go, don't try to destroy the game once you've left. Keep the break clean, and don't hassle those that are still enjoying the game. If it is a big problem and the whole group is suffering, then the group will find something new soon enough, but you don't make your case better by poking and needling at it the entire time.
The other reason for a clean break? I joked about it in the beginning, but some people take their gaming very seriously. It is actually a disclaimer we give to new people in our friday group, "We take our gaming seriously. So if you can't regularly commit to being here, don't ask to join." We don't mind teaching people, we don't mind trying new things, but we still take the game seriously. Other people get very attached to aspects of their game or their characters. Even others, will wonder why you left or feel betrayed that you even did.
Point is, be prepared for some level of an emotional response. This is where being direct, but polite, is important. People's feelings are going to get involved, and contrary to popular belief the social aspect of an RPG group is not "just a game". So, be respectful, and be prepared. That is all you really can do.