Context. High-contrast imaging surveys for exoplanet detection have shown that giant planets at large separations are rare. Thus, it is of paramount importance to push towards detections at smaller separations, which is the part of the parameter space containing the greatest number of planets. The performance of traditional methods for the post-processing of pupil-stabilized observations decreases at smaller separations due to the larger field-rotation required to displace a source on the detector in addition to the intrinsic difficulty of higher stellar contamination. Aims. Our goal is to develop a method of extracting exoplanet signals, which improves performance at small angular separations. Methods. A data-driven model of the temporal behavior of the systematics for each pixel can be created using reference pixels at a different positions, on the condition that the underlying causes of the systematics are shared across multiple pixels, which is mostly true for the speckle pattern in high-contrast imaging. In our causal regression model, we simultaneously fit the model of a planet signal “transiting” over detector pixels and non-local reference light curves describing the shared temporal trends of the speckle pattern to find the best-fitting temporal model describing the signal. Results. With our implementation of a spatially non-local, temporal systematics model, called TRAP, we show that it is possible to gain up to a factor of six in contrast at close separations (<3λ∕D), as compared to a model based on spatial correlations between images displaced in time. We show that the temporal sampling has a large impact on the achievable contrast, with better temporal sampling resulting in significantly better contrasts. At short integration times, (4 s) for β Pic data, we increase the signal-to-noise ratio of the planet by a factor of four compared to the spatial systematics model. Finally, we show that the temporal model can be used on unaligned data that has only been dark- and flat-corrected, without the need for further pre-processing.