Every RPG out there has an initiative system. It is a necessary evil when it comes to playing games like Table Top RPGs. Necessary because when things get hectic in the game, there needs to be a system that figures out who can talk and whose actions resolve when. Evil because by their nature initiative systems take what is a very exciting and chaotic sequence of events and slows it down while chunking it into very orderly segments or bites.
However, while every RPG comes with a pre-packaged initiative system, I can't think of a single game where how the initiative system works is of fundamental importance to the system. And with the exception of 7th Sea 2nd Edition I have yet to see a game with an initiative system that can't be taken and used in other games. I could be off on both of those - my experience is limited - but in most games - especially the mainstream ones - you can change initiative systems with relative ease.
That said, let's talk about initiative systems and why you may want to change from the default in your game.
Static vs. Dynamic
The first thing I want to talk about is initiative systems fall into two broad categories: static and dynamic. In a static initiative system, once turn order is determined it holds true for the entirety of the scene. In a dynamic system, initiative changes from round to round.
The benefit of a static system is that you don't need to worry about book keeping for initiative after it is done. You have an order and you just run through that. Players know when they go. They know when monsters go. And that knowledge can be both a limitation and a boon for tactical decisions.
The benefit of a dynamic system is that it keeps things a bit more mixed. Just because someone went first in round 1 doesn't mean they go first in round 2. This means you need to pay attention - which can keep players engaged - and it also means that the options for easy teamups you had in the first round may not be there the second, meaning you have to be more creative. On the downside, dynamic initiative systems do require more book keeping which can slow people down.
Everyone Rolls For Themselves
The most common initiative system I've seen is one where every character or agent rolls for themselves. Each player rolls their characters initiative. Each monster has its own initiative. Order is determined from high roll to low roll (or vice versa) and you go through that list. In a dynamic system like this you roll every round, or some other math is done at the end of the round to see where people move (i.e. wound penalties, action timers, etc.) In a static system you make an order and just go through it over and over again.
This is the bread and butter for initiative. It works. However, it also makes for a lot of rolling at the beginning of combat - especially when you have a larger table of players or a large number of monsters/npcs involved in a combat.
Player Slots, Ally Slots, Enemy Slots
I first saw a system like this in FFG's Star Wars games. How it works is everyone rolls initiative separately, but you aren't getting an initiative slot for your character but rather for your side. From round to round the PCs can choose which PC goes in which slot in the initiative order - provided they have yet to act that round. Allied NPCs and Enemy NPCs do the same with their own groupings.
This can work well for more cinematic or tactical combats because it lets players really choose when they want to go if they have something to contribute. It is easier to coordinate a combination move when you can have the necessary pieces go in the slots available for it. It is also possible to give someone two turns in close proximity - going near last one round, and then near first the next - in order to unload a big attack and scoot back to safety.
The pitfalls come in two places: first, you have to track who has gone and who has not which is more paperwork. Second, until the group gets comfortable with it - and some groups never do - there is often a lot of hesitation as no one wants to take a slot someone else may need leading to pauses as people look around the table and go "does anyone want to go now? no? I guess I could do something."
Groups Go At Once
A variant of everyone rolls for themselves, with this each group has one initiative roll. When that initiative turn comes up the entire group goes. This basically leads to "All the PCs go" and then "All the enemy NPCs go" in turns. This can work out, and it definitely gives power to wanting to go first since that can be a hell of an alpha strike to take down a priority target. Still it can lead to a lack of being able to 'interact' with the other team. You and all your allies go at once. They all go at once. If someone gets in trouble, there's no real way to help or save them. They need to survive the entire enemy turn.
On the other hand, this has very little paperwork. You can determine initiative in most combats with a coin flip. It is also easy to make combination attacks.
End Turn By Choosing Who Goes Next
One of the favorite systems I've seen for Initiative but have not had a chance to play with a lot is one where you roll to see who goes first, but then at the end of each characters turn they choose who goes next. The only rule is that you can't choose someone who has gone this round.
On the down side, this means that you need to track who has had a turn this round. Also some players may feel it pointless to be rolling initiative when only the high roll matters. Beyond that, you have a lot of upsides. It is easy to combine actions because if I want to combine with you, I can just give you your turn when I'm done. The 'abuse' case of all the PCs go at once is also solved because if the PCs can't win the fight in one turn, they leave themselves open to all the enemies getting two turns before they get to go again.
Special Consideration: Declare Up, Resolve Down
In older editions of Legend of the Five Rings, and a couple other RPGs, initiative and turn order had two phases. In the first phase you declared from lowest roll to highest roll what you intended to do. In the second phase you resolved those actions from highest roll to lowest roll. The idea being that the people with high initiative got to know what everyone acting after them was going to do before they chose their action. This rewarded them with not only getting to resolve their action first, but superior knowledge of what was going on.
I like the idea here, but in practice it has a couple things to consider. First, what happens when the person with low initiative declares they're attacking NPC 1, only NPC 1 dies - or otherwise becomes untargetable - before their turn? Do they lose that action? If so, that feels bad. If not, what was the point of declaring that action when it wasn't going to be held to? Second, it nearly doubles the amount of time to resolve a turn, and can be confusing for people who think they should be rolling when they declare their turn.
Which Is Right For Your Game?
So which is right for your game? That depends on you, the game, and your group. If you're bored or unhappy with how initiative is currently working - or just want to try something different - talk it out with your group.
Remember though, how you do initiative will impact how combats play out. It opens tricks on both sides for how things can be done, while imposes other restraints. Find one that works for you and your group.