Why are some galaxy groups pervaded by a hot X-ray emitting intracluster medium, whilst others have no detectable X-ray emission? Is the presence of hot gas a reliable indicator of dynamical maturity, and can some virialised groups contain little or none of it? What are the main differences between samples of groups selected in the X-ray and optical bands? We address these questions by studying 232 optical spectroscopically selected groups from the Galaxy And Mass Assembly (GAMA) survey that overlap the XXL X-ray cluster survey. X-ray aperture flux measurements combined with GAMA group data provides the largest available sample of optical groups with detailed galaxy membership information and consistently measured X-ray fluxes and upper limits. A sample of 142 of these groups is divided into three subsets based on the relative strength of X-ray and optical emission, and we see a trend in galaxy properties between these subsets: X-ray overluminous groups contain a lower fraction of both blue and star forming galaxies compared with X-ray underluminous systems. X-ray overluminous groups also have a more dominant central galaxy, with a magnitude gap between first and second ranked galaxies on average 0.22 mag larger than in underluminous groups. Moreover, the central galaxy in overluminous groups lies closer to the luminosity-weighted centre of the group. We examine a number of other structural properties of our groups, such as axis ratio, velocity dispersion, and group crossing time, and find evidence of trends with X-ray emission in some of these properties despite the high stochastic noise arising from the limited number of group galaxies. We attribute the trends we see primarily to the evolutionary state of groups, with X-ray overluminous systems being more dynamically evolved than underluminous groups. The X-ray overluminous groups have had more time to develop a luminous intragroup medium, quench member galaxies, and build the mass of the central galaxy through mergers compared to underluminous groups. However, an interesting minority of X-ray underluminous groups have properties that suggest them to be dynamically mature. We find that the lack of hot gas in these systems cannot be accounted for by high star formation efficiency, suggesting that high gas entropy resulting from feedback is the likely cause of their weak X-ray emission.