Fundamental interactions of nature are described by gauge theories whose dynamical variables are matrix valued. Perturbation theory fails in describing many interesting features of these theories such as confinement. An alternative approach to this problem is the large dimension ( N ) limit of 't Hooft, which is still not solvable, and further approximation methods are needed. One such method is the variational approximation. We show that averaging over gauge degrees of freedom creates an entropy which has to be included in the variational principle. Moreover, we identify this entropy with the notion of free ‘noncommutative’ entropy of Voiculescu. In the case of quantum mechanical matrix models we show that a similar term in Hamiltonian is generated for similar reasons and identify it with Voiculescu's free ‘noncommutative’ Fisher information. We apply our variational principle to exactly solvable cases and obtain good agreement. We also show the generality of our methods by applying them to some systems that are not exactly solvable.