Multiple-choice questions (MCQs) evaluate factual knowledge in medical education and have a high reliability, if performed appropriately. However, many MCQs contain formal errors leading to reduced validity. The authors developed a Web application capable of recognizing and eliminating five frequent contraindicated practices in MCQs: negative stem, unfocused stem, cueing words, longest item = right item flaw, and stem/item similarities. The authors used simple string algorithms and dynamic comparisons with keywords. The system was successfully validated with a sample of approximately 800 continuous medical education (CME) questions, showing that our system automatically detects 60% of all formal didactic errors. Flaws not detected by the software can easily be avoided using quick manuals on item wording or clear instruction to the authors. The authors conclude that it is feasible to improve the quality of MCQs by designing a Web application that is capable of detecting common flaws by simple string operations.