If dxtory and lagarith lossless codec doesn't satisfy you no other record will. Set it to 720p/1280x720 and 30FPS. Maybe it's not the recorder that's the issue? Are you recording to a separate drive? what is your write speed in dxtory?
+1 Shadow play if you have compat Nvidia card. I have it on every game (just in case I do something EPIIIIIIC) and it doesn't impact my performance at all on Med quality (i5 3570, 8gb ram, gtx660ti)
@llchuckwagonll, I'm wondering what you mean when you say that ShadowPlay does not give the same quality as Lags & dxtory? I haven't tried the latter ones, but I use ShadowPlay a lot, and the recordings are simply perfect. I actually cannot see any difference between the real in-game visuals and the recording, so what exactly is it that you are not satisfied with?
Shadowplay does not record lossless, and thus the quality is always compressed. If you zoom in videos etc, you'll notice that it does not scale all that well, for example. It looks good running at the same resolution it was recorded on, yes, but NOTHING beats lossless quality. Even if you can't properly see it just playing back the videos.
Aha, thanks @D2ultima! I get it now, though I have to say that the minor quality loss induced H.264 encoding hardly is worth mentioning when we are talking about in-game recording of an FPS game. I mean, it's not like you are going to publish the raw loss-less recording anywhere anyway, is it? With my setup, recording Titanfall in-game in native screen resolution (2560x1080), 60Hz (which is what I run the game in) and 50 Mbps, a 20 minute recording (which is the size that I've set the recording buffer to), the resulting Mp4 file is roughly 4 GB large. A bit size-y for YouTube I'd say... ShadowPlay is free, you can record any game with it, and doesn't (well - shouldn't, but it's still in BETA) affect your FPS. But I suppose it comes down to personal preferences.
That's all well and good, but for certain video editors they want all the quality they can get. When they've done all their editing, then they compress. My friend will work with 20GB large 5 minute video files when he's running it from Sony Vegas lossless render to Adobe After Effects, then he'll render that once again losslessly and then possibly toss it into Cinema 4D for some more effects, then back to Sony Vegas it goes where it renders in a lossy codec and then he'll toss it into Handbrake for good measure afterward. To me, I just toss on Playclaw and get a 9.47GB file for 13 minutes of sleeping dogs at 1080p/60fps XD. That's with 75% quality, I could tone down the quality for less filesize too. Also, you could just grab it with OBS which is magical if you know what your settings are for the game you're recording, but again, when you want to zoom in and pan and add minuscule effects to videos, lossless codecs are the way to go. Nobody ever really uploads lossless videos though, it'd take far too long and nobody could watch it XD.