Sensor fusion / Kalman filter was indeed the right direction to look in, but I never got the time to try a reference implementation unfortunately. Barometer + accelerometer might've worked for much larger discrepancies in height but I don't think the data is useful outside of sensor fusion.