Background Though many trials had examined the effectiveness of taking insulin with or without oral agents, there are limited real-world data, particularly among patients with type 2 diabetes mellitus (T2DM) in the resource limited settings. This study aimed to examine level of glycemic control among patients with T2DM after initiation of insulin and factors associated with poor glycemic control. Methods An analysis of retrospective medical records of patients with T2DM who initiated insulin due to uncontrolled hyperglycemia by oral agents was conducted from 2015–2020 in the University of Gondar Comprehensive Specialized Hospital. Difference in median fasting plasma glucose (FPG) before and after insulin initiations was examined by a Wilcoxon signed-rank test. Kruskal Wallis test was performed to explore difference in the median level of FPG among treatment groups. A logistic regression model was also used to identify associated factors of poor glycemic control after insulin initiation. Statistical significance was declared at p < 0.05. Results Of 424 enrolled patients with T2DM, 54.7% were males and the mean age was 59.3±9.3 years. A Wilcoxon signed-rank test showed that there was significant deference in FPG before and after insulin initiation (P < 0.001). A declining trend of blood glucose was observed during the 1-year follow-up period of post-initiation. However, majority of the participants did not achieve target glucose levels. Participants who had higher FPG and systolic blood pressure (SBP) before insulin initiation were found more likely to have poor glycemic control after insulin initiation. Similarly, patients who received atorvastatin compared with simvastatin were found to have poor glycemic control in the post-period of initiation (P = 0.04). Premixed insulin was associated with a lower likelihood of poor glycemic control than neutral protamine Hagedorn (NPH) insulin (P < 0.001). Conclusion Following insulin initiation, a significant change in glycemic level and declining trend of FPG was observed during a 1-year follow-up period. However, the majority of patients still had a poorly controlled glycemic level. Appropriate management focusing on predictors of glycemic control would be of a great benefit to achieve glycemic control.