The most commonly used antenna for the 5.8Ghz FPV is the Cloverleaf antenna, due to its onmi-directional radiation pattern. Helical and Patch antenna are directional antenna and they provide longer range.But it does not do very well when you fly to the sides or behind it.
5.8Ghz Directional Antenna in General
Basically the higher gain the antenna, the more focus the beam is. It allows the signal to reach further, but the trade off is you get narrower beam (angle of reception). Personally I would not just use a directional antenna alone because you get very limited range on the sides, and behind it. Especially for quadcopters, which usually fly around the pilot. I would probably use a diversity system that has multiple directional antennas to increase the angle of reception, or couple with a onmi-directional antenna.
When flying behind objects, signals could be blocked and you might lose signal even with high gain directional antennas. But if the signal is only partly weakened by the object, not blocked completely, helical/patch antenna should still help pick up that signal, which onmi directional antenna would normally struggle to do.
Anyway to get the best possible penetration ability, lower frequency is preferred, as they are better at penetrating or to go around obstacles.
Helical antenna are great for preventing multi-path, suitable for both 2.4GHz and 5.8GHz. The Helical antenna has a wider band reception than patch antenna. Maximum efficiency I have seen is about 80% with a 3-4 turn helical and drops to about 60% at 12 turns which is more than acceptable.
The helical has sidelobes that allow you to fly to the sides and behind it (Not too far, but still better than patch).
This is a 5.8GHz 12dbi Helical antenna. Helical for 5.8GHz FPV system is quite small, and can be mounted on the FPV goggles directly. But the more turns you have, the longer it gets, so it’s not as compact as the patch antenna.
It’s not easy to make good helical antenna, because it needs a lot of tuning, and accurate cutting. So buying from well known professionals is probably the best choice.
Patch antennas are not as good with multi-path rejection compared to Helical.
Generally for patch antenna, the gain usually goes higher than 12dbi, so the beam is not as flexible as the helical (where you can choose fewer turns to have a wider angle with helical).
Patch antennas are great option if you need some light weight, compact solution.It would be ideal for a diversity setup but if you only fly close to yourself like in a park or football pitch then cloverleaf style antenna is better.
Inside of the patch antenna looks like this.
Other good directional antenna
The Crosshair is another good directional antenna for video receivers. Although I have not personally tested this, I was told it has a wide beam (it could reach 115 degrees which is equivalent to a 3 turn helical), yet has high gain which gives you more range. But the signal degrades quickly on the sides, and behind compared to the helical.