I think it's safe to say that anyone who is passionate about tabletop games has at some point tried to design their own. And anyone who did try knows it's not in any way easy. But what goes into the rule system of a game? How do you make it fun? How do you make it balanced? Come to think of it, what do "fun" and "balanced" mean, anyway?
I had to answer these questions while designing a system of rules for Dungeon Romp. Which was a choice in itself, as the more obvious option was to go with an existing system. But it wasn't the right one.
My first encounter with RPGs was via a second hand AD&D2 Player's Handbook. It blew my mind: here was something not unlike my beloved gamebooks, but which you could play together with friends, and not as opponents, either. But years later, when I had the opportunity to play for real, the experience was disappointing. We kept forgetting rules, and those we did remember to follow felt like a chore. Going freefrom turned out much better, both face-to-face and over the Internet... except sometimes — just sometimes! — the need to roll some dice was overwhelming.
If I had to prepare a tabletop game nowadays, I would probably go with Risus, as it's just about the simplest RPG rule system not aimed at children. But you still have to learn all of it up front, and if you ever need to extend the rules, well, let's just say the system doesn't lend itself naturally to that.
Indeed, that's what bothers me about all the RPGs I know. They're either too detailed, too coarse, or too weird. Fudge is explicitly configurable, but you need to select the level of detail up front. I.e. before you know how much detail you'll need. That, and almost all of them forbid commercial reuse. Which is a problem if you want to distribute a derived work under an open license, let alone sell it.
So it happens that I long wanted to create my own RPG rules that would, well, be my own and support incrementally adding detail. But I had no incentive to develop a system before Dungeon Romp showed on the radar.
My yet unnamed RPG system
Trick question: what's the smallest amount of attributes you need for RPG combat?
Answer: technically, none; you could simply toss a coin after all. But that removes player agency, and then what you're left with is Snakes and Ladders. Or a slot machine.
For Dungeon Romp, I started with only two attributes: Agility, which determines both attack and defense, and Stamina, which determines how much damage you can take. Right there, you have an opportunity for customization: do you want a frail character who is hard to hit yet strikes true himself, or a big resilient character who survives fights rather than winning them?
There is also Senses, which determines how far you can see, but it has little connection with the rest. And then there's another, special attribute.
Non-trick question: what's the essential difference between 1D20 and 3D6? I hope you didn't answer "the extra range".
The answer is, of course, that the latter smooths out the randomness, making it far more likely to roll a 10 or 11 than a 3 or 18. Whereas with the former, your chance at a critical hit or fumble is exactly the same as that of rolling a boring average result. Which not only isn't plausible, but makes combat way too arbitrary.
That's why I wanted to have a dice pool system right off the bat. Except there's a little problem with that, too: someone with 5 dice in an attribute with consistently trounce someone with 3, even though the respective ranges overlap for almost two-thirds of the way. That's because the more dice you have, the more uniform rolls you get.
The solution was to give each character a Dice Size attribute. Which is essentially a scale, except it doesn't even try to be realistic. Imagine a D10 human trying to fight a D500 dragon! That, and concern for the future made me stick with real-life dice sizes. The computer grants the game designer absolute freedom; you want to use it wisely.
As an aside, there's something to be said for sticking with "physical" rules in computer games. Compare, for example, the AD&D2 rules as implemented in the Infinity Engine games with the custom system in Fallout 1 and 2. The former must have taken a lot of work to implement, especially as it had to be done all at once, but the result is dynamic and interesting to play (once, you know, the player is freed from keeping track of all the minutiae); whereas the latter just feels clunky, even though it's made to take advantage of its native medium.
Anyway, once I added the dice size, combat became surprisingly easy to balance. For example, 3D8 will always beat 5D4 in the long term, while clearly suggesting that the smaller creature is much stronger and faster relative to its size. Except when they don't, but then you get plenty of advance warning. And once I had the basics down, it was easy to add more attributes: Muscle, which gives the stronger opponent a damage bonus (and soaking, in defense), and Speed, which determines how many actions different mobs can take in a row, relative to each other.
There is more, of course. You only roll dice twice for each combat turn — once for attack, once for defense — and the margin of success matters. Bonuses from armor and weapons are either additive or (rarely) mutiplicative, but always fixed, and so on. The rules are rich but simple; they can be extended easily, and (re)balancing is a snap. All in all, I'm happy with the result and can't wait to see where it takes me, both with Dungeon Romp and other future games.