In general, using unmatched fins tends to rock the boat back and forth as the greater lateral thrust of the larger fin is felt. You can get this effect to varying degrees by using one fin only, using mismatched fins or even setting the clews grossly differently.
Having said that, it is
possible, to successfully combine a stiffer fin with a more flexible fin (as for example, combining an original Turbofin with a current production Turbofin). The stiffer fin (having less blade twist) is set in the rear position and that clew outhaul is opened up to increase the pitch. The idea, like with the contra rotating props referred to above, is to take advantage of the altered slipstream created by the leading fin.
With the Mirage Drive, this can generate a very small gain but the old style fins are now hard to come by and the gain is probably not worth the trouble for most. I do have a set-up like this however -- need all the help I can get!