Come on, let's blame piracy !
12 years of DNF vaporware, I am supprised they didn't close earlier. Thier last game PREY wasn't even finished by them, they handed it off to some 3rd party developer to finish.
Prey by 3D Realms ? What ? Prey was made by Human Head Studios, which consists mostly of people who left Raven Software after developing Heretic and Hexen (some of my favourites of all time). Since their departure, Raven hasn't made a single original game, they just milk old cliches like Star Wars, Star Trek etc. And they completely abandoned fantasy games.
Human Head also made a great game Rune (highly recommended).
Anyway, back to topic. I think Duke Nukem 3D was somewhat overrated. Once the novelty of shooting clocks on the walls wore off, it was a pretty average game. It didn't have the polished visuals of Doom (I'm talking mostly about boring style and monsters of DN3D. Pigs in police uniforms. UFO guys. Facehuggers or flying octopi. Nothing interesting or original). I think what set DooM apart is that it really had good visuals and sound, and I'm talking about quality of textures, art direction etc. It really shows when you run early alphas and press demos of DooM 1. Textures suck compared to what you remember from original DooM or DooM II.
I like Hexen 1 much more than DN3D. The two games had in common that they were doom-like games with extra gimmicks, (Hexen used, in fact, a modified DooM engine). But art direction and style of Hexen was much better in my opinion, and it had very dark atmoshpere. Hexen had all the gimmicks DN3D had (looking up and down, jumping, summoning allies instead of holoduke, freezing enemies to solid ice, setting them on fire, earthquakes, special effects, slick ice floor, water currents, submering enemies) but better style. And DooM engine being open-source it enabled Hexen to look very well even by today's standards. A screenshot from a modern Hexen source port...
http://archive.doomsdayhq.com/images/intro_jhexen.jpg
Unfortunately Hexen had some bad design decisions, most notably hunting hidden switches that would open something somewhere without telling you where. Hexen levels, while pretty and interesting, were also confusing to navigate and had lower replay value, unlike those of DooM (which are usually winnable even if you start without any weapons on Ultra-Violence skill ! DooM II levels were great for competitions with friends - who can win this level first, etc)
In short, I think Duke had a pretty small window of opportunity between DooM and Quake, and it wasn't unique in gimmick department either.