I think my hosting is quite bad (OVH).

Sorry to hear you had problems, I have done a search for possible problems with wp object cacheing and found this :

Object cache may cause performance penalty when the available memory is low and too many Apache processes are allowed. With caching enabled each Apache process (or Thread), serving a WordPress page request, caches at most 124 php objects which can increase memory consumption.
As a result kswapd (swap manager) comes into play and at high loads it tends to swap too often and ends up consuming most of the CPU cycles itself instead of allowing actual processing.
So if you are on a low end machine with low RAM or if you are sharing the machine with lots of other users then you can experience performance issues. However you can easily solve it by either increasing the memory or reducing the maximum number of Apache processes you serve. In short object cache can be used to improve your performance.