I'm having the same problem, and i can confirm that it is shaders, since i'm not using any other mod along with them besides optifine. I don't know how to fix the problem, though, since the issue isn't that it's actually changing the settings, as you said, it's as if you were given the slowness effect. It also seems to be a chunk border thing, though i can't say for sure because it only happens rarely, and not always in the same place.
Damage indicators
NEI (with chicken core)
Lite loader
Treecapitator (with core)
Voxel map with radar and cave for lite loader
Optifine
Shaders core
Spices of Life
And sometimes when i play the FOV changes like i get slowed.
Or back to default which is 70 and im at 95
And i am not getting slowed by anything it just suddenly goes to a lower FOV and then after a random time it goes back
it happens both in SSP and SMP