It usually has to do with how the renderer is written. The renderers of each engine will handle graphical pipelines in their own particular way. As Plainview already stated, the Doom 3 engine had a very particular look across Prey, Doom 3 and Quake 4. But even back in the 1990s when there were less graphical effects being drawn, you could still distinguish between Quake 2 (Soldier of Fortune, SIN, Anachronox), The Dark Engine (Thief, System Shock 2, Thief 2) and Unreal (Deus Ex, Rune, X-com: Enforcer) pretty easily.
Some developers will rewrite portions of the renderer, like Rocksteady using Unreal 3 to make Arkham Knight, which looked on par with Unreal Engine 4 games. On the original Xbox, Ion Storm developed a lighting model within the Unreal 2 engine very similar to Doom 3 for use in Deus Ex: Invisible War and Thief: Deadly Shadows, giving those games a very distinct look relative to the engine. The most famous example I could give you would probably go to Valve's offshoot of the Quake Engine with their original Source engine. Half-Life, it's expansions and the hundreds of available mods all have that "GoldSource look." An interesting tidbit you might not know is that Duke Nukem Forever is running on a modified version of Unreal 1, not Unreal 2 or 3.
Anyway, most Unreal 3 games looked very similar in how they were rendered. For instance Singularity, Bioshock and Arkham Asylum displayed how they were built around the strengths of the engine. A lot of last-generation games were built on UE3, so we got to see a larger set of art styles using the renderer, which was very interesting and probably a first-time occurrence for this industry. While a lot of games used Quake and Unreal for example, they were typically used for developing strictly action games with a similar aesthetic.
To tune into this phenomenon yourself, you can play around with Unreal 4, CryEngine, Unity, and any other modern engines that interest you to get a feel for the output of their differing shader methods.
There are games being built on Unreal 4 today, such as Graven, that are going for a "Quake look" on modern tech. Until they replace the lighting model with something more primitive, reduce the palette to 256 colors including fullbrights, and go back to vertex-manipulated animations, it's just not going to look like a Quake engine game. Wrath: Aeon of Ruin, using a Quake Engine source port called "Darkspaces" very much retains the "Quake Engine Look."