Activity recognition for human behavior monitoring is an important research topic in the field of mHealth, especially for aspects of physical activity linked to fitness and disease progress, such as walking and walking speed. Sensors embedded into smartphones recently enabled new opportunities for non invasive activity and walking speed inference. In this paper, we propose a data fusion approach to the problem of physical activity recognition and walking speed estimation using smartphones. Our architecture combines different sensors to take into account practical issues arising in realistic settings, such as variability in phone location and orientation. Additionally, we introduce a novel automatic calibration methodology combining accelerometer and GPS data while walking in unconstrained settings, in order to reduce walking speed estimation error at the individual level. The proposed system was validated in 20 participants while performing sedentary, household, ambulatory and sport activities, in both indoor laboratory and outdoor self-paced settings. We show that by combining accelerometer and gyroscope data, smartphone location can be distinguished between the two most commonly used positions (bag and pocket), regardless of phone orientation (97 % f-score). Location-specific activity recognition models can significantly improve activity recognition performance (p = 0.0010 <; α), especially helping in distinguishing activities involving similar motion patterns (91 % f-score overall, improvements between 4% and 11 % for walking and biking activities). Our proposed method to personalize walking speed estimates, by automatically calibrating walking speed estimation models during a short self-paced walk, reduced walking speed estimation error by 8.8% on average (p = 0.0012 <; α).