Phytochrome proteins control the growth, reproduction, and photosynthesis of plants, fungi, and bacteria. Light is detected by a bilin cofactor, but it remains elusive how this leads to activation of the protein through structural changes. We present serial femtosecond X-ray crystallographic data of the chromophore-binding domains of a bacterial phytochrome at delay times of 1 ps and 10 ps after photoexcitation. The data reveal a twist of the D-ring, which leads to partial detachment of the chromophore from the protein. Unexpectedly, the conserved so-called pyrrole water is photodissociated from the chromophore, concomitant with movement of the A-ring and a key signaling aspartate. The changes are wired together by ultrafast backbone and water movements around the chromophore, channeling them into signal transduction towards the output domains. We suggest that the observed collective changes are important for the phytochrome photoresponse, explaining the earliest steps of how plants, fungi and bacteria sense red light.