Healthcare systems are challenged by increasing costs. Digital technology can help to combat this trend. Evaluation of these technologies is uncommon or incomplete. Scholars have called for a standardized and holistic evaluation. We provide a synthesis of an online panel on medical informatics (MI) and stipulate a discussion on new guidelines for medical informatics project evaluations. The panel consisted of presentations and a discussion. The presentations gave the participants an overview of evaluation methods currently used in different medical informatics domains and their shortcomings. The presenters highlighted new evaluation methods such as a roadmap for economic analysis of eHealth projects and the German Digital Healthcare Act methods. Participants discussed the shortcomings of RCTs and methods that need to be included in eHealth evaluation and called for new evaluation methods. The discussion showed weaknesses of the currently used methods and underlined the need for a new, holistic evaluation standard for MI.