In my opinion, the surest sign that a GM owns a system - or at least a particular campaign - and knows exactly what they're doing, what the PCs are capable of, and what the PCs will think of, is when their Threat Assessment is spot on. Fights are exactly as challenging as they want them to be, and while the dice may swing a fight one way or the other, in general the things they want to feel challenging are challenging, and the ones they want to be easier are easier. The question is, how do you get good at assessing threat?
Nothing Is Easy
Unfortunately, there is no easy way to get good at threat assessment for a system. It is going to take time, and a lot of trial and error. Every system has its own nuances and little tricks that you can pull to make things harder or easier, and you won't be able to master them without first employing them a few times. This problem gets even harder when you consider that every group of PCs will have different abilities, giving them different strengths and weaknesses that you have to deal with.
Essentially, what I am saying is that there isn't really a quick route to getting good at it. Hell, I've been gaming for nearly 20 years now, and GMing for about fifteen of those years, and the only system I think I'm actually really good at judging threat assessment on is Legend of the Five Rings third edition. A game which I Head GMed a game of over 100 players for two years, along with an additional three to four years of other GMing and playing experience at all walks of power levels, including blended scenarios where there were high and low level characters in the same group.
Don't Trust Challenge Rating
Some systems come pre-packaged with a built in Challenge Rating system, where they let you know about how tough an enemy is to beat. It will give a stat, like say a "Challenge Rating 17" meaning that the creature should be a challenging fight for 4-6 PCs of around level 17. While this is a handy tool to narrow the field down, I'd recommend you not trust it completely. Why? Well, because like I said before, every group of PCs is different. Even given the same stat blocks, each group will play those characters differently, choose different spells, and do other things differently in a fight. Meaning that what is good for the book designer, may not work for you.
As an anecdotal story, in a D&D 3.5 game I was in a few years back, our group of PCs absolutely trounced a combat that was a CR for several levels higher than us. The GM had even put that fight in the way because of how quickly we'd been mowing through the opponents set in front of us. Then, a session or two later, we came upon what was supposed to be an easy fight. A single creature with a CR for about four levels lower than us. The creature mopped the floor with the entire party, wiping the group save for the elven archer who ran away. Why? Because that creature used psionics and our group was ridiculously weak to psychic stuff. The fight would have been easy, except our group wasn't built or equipped to handle that kind of a problem, and the GM hadn't taken that into consideration when putting the fight down there. Not even really the GM's fault either, considering the books agreed with him on how hard the fight should have been.
So, there's been a lot about what to watch out for, and what to expect, so how do you actually get good at it aside from trying again constantly? Well, in my opinion the best way to do it is to start small. Starting with the first session with every new campaign, try to start off with some easy fights. These will let the players get used to what they can do, and let you get used to what they will do. Then, as the game goes on over the next couple of sessions, slowly ramp up the difficulty on the fights until you think you've hit the sweet spot. The place where the fight could very easily have gone either way if the dice had come down differently.
Now, you need to pay attention while doing this as well. If the PCs are winning constantly because they've hit a string of really good rolls, then you should move the difficulty slider up a bit more slowly. Luck can't last forever, and when the players start rolling poorly - or at least more in line with a bell curve - you don't want to wipe them out with an enemy group balanced against the fact that every other shot is a critical success.
The final bit of advice for this piece, use smart enemies while ramping things up. Smart enemies give you a tool that the mindless beasts just don't have. They can act intelligently. Meaning,t hey can hit the group with a plan. "We go in, soften them up, and fade fast.", or "We go in, hit them hard, grab the mage's spell book, and run". This way, if you go a bit over board with threat assessment on accident, you can still salvage the situation, hand the players a loss, and start a side quest, all without having to wipe the group out.
Hell, do it right, and they'll never even know they were supposed to mob the floor with that group of bandits that abducted the thief character.