Analog electronic and photonic crossbar arrays have been emerging as energy‐efficient hardware implementations to accelerate computationally intensive general matrix–vector and matrix–matrix multiplications in machine learning (ML) algorithms. However, the inevitable nonuniformity in large‐scale electronic and optoelectronic devices and systems prevents scalable deployment. Herein, a calibration approach is reported that enables accurate calculations in crossbar arrays despite hardware imperfections. This approach is experimentally validated in a small‐scale free‐space photonic crossbar array based on cascaded spatial light modulators and demonstrated the scalability and universality of this approach in various large‐scale electronic and photonic crossbar arrays. The improved performance of calibrated crossbar arrays in an ML model inference is further demonstrated to classify handwritten digital images.