The optimization that actually worked meanwhile was the fact I was launching threads during my compute dispatch just to discard them afterward in the shader code.
Now instead I launch exactly the number I need and compute a better index for processing my geometry.
So just helping the GPU schedule things better gave me 0.04ms saving on around 140K meshes (went from 0.1ms to 0.065ms). That's on my beefy GPU, I presume on my old laptop this will be even better.