Abstract Background Patient-derived organoids are invaluable for fundamental and translational cancer research and holds great promise for personalized medicine. However, the shortage of available analysis methods, which are often single-time point, severely impede the potential and routine use of organoids for basic research, clinical practise, and pharmaceutical and industrial applications. Methods Here, we developed a high-throughput compatible and automated live-cell image analysis software that allows for kinetic monitoring of organoids, named Organoid Brightfield Identification-based Therapy Screening (OrBITS), by combining computer vision with a convolutional network machine learning approach. The OrBITS deep learning analysis approach was validated against current standard assays for kinetic imaging and automated analysis of organoids. A drug screen of standard-of-care lung and pancreatic cancer treatments was also performed with the OrBITS platform and compared to the gold standard, CellTiter-Glo 3D assay. Finally, the optimal parameters and drug response metrics were identified to improve patient stratification. Results OrBITS allowed for the detection and tracking of organoids in routine extracellular matrix domes, advanced Gri3D^®-96 well plates, and high-throughput 384-well microplates, solely based on brightfield imaging. The obtained organoid Count, Mean Area, and Total Area had a strong correlation with the nuclear staining, Hoechst, following pairwise comparison over a broad range of sizes. By incorporating a fluorescent cell death marker, intra-well normalization for organoid death could be achieved, which was tested with a 10-point titration of cisplatin and validated against the current gold standard ATP-assay, CellTiter-Glo 3D. Using this approach with OrBITS, screening of chemotherapeutics and targeted therapies revealed further insight into the mechanistic action of the drugs, a feature not achievable with the CellTiter-Glo 3D assay. Finally, we advise the use of the growth rate-based normalised drug response metric to improve accuracy and consistency of organoid drug response quantification. Conclusion Our findings validate that OrBITS, as a scalable, automated live-cell image analysis software, would facilitate the use of patient-derived organoids for drug development and therapy screening. The developed wet-lab workflow and software also has broad application potential, from providing a launching point for further brightfield-based assay development to be used for fundamental research, to guiding clinical decisions for personalized medicine.