BackgroundOnline video-based learning is more common in higher education. Investigating students' viewing behaviors while watching online video lectures is essential for instructors to understand their learning status so that the course content, structure, and media selection can be improved continuously. The current study identified the engagement level of the learners based on their online video-watching behaviors, and tested the correlation between the engagement level and learning outcome.MethodsThe action logs of watching online video lectures in 2020 Spring Pharmacology of the 4th-year medical students of the 6-year course and their feedbacks by questionnaires after each exam during the semester were provided anonymously. The data were analyzed and visualized for an efficient way to comprehend and interpret. To define the student's engagement level in his or her video-based learning journey, three viewing criteria, “Completion,” “Pausing,” and “Repeated watching” were identified. We evaluated the association between the engagement level and the students' learning outcomes, including their learning satisfaction, knowledge acquisition progresses based on assessment results, and the grades measured by the instructors.ResultsThe graphs and the charts demonstrate whether the students allocated enough time to finish the video lectures (completion), paused for a while, then resumed the video (pausing), or replayed the specific sections of video content (repeated watching). The engagement level with video lectures, evaluated by pre-defined thresholds for “Completion,” “Pausing,” and “Repeated watching” had a positive correlation with the learning outcomes.ConclusionsWe suggested that an engagement dashboard containing real-time visualized information on students' online video-watching behaviors can be developed to help instructors to monitor students' learning progress and improve teaching in a timely fashion. It can also help each student to re-feel the stimulation of peers, prompt self-monitoring, improve their learning attitudes and disciplines for better learning outcomes. This innovative way of assessing student's engagement during online video-based learning can also be used for quality assurance purposes.