CPU cycles are cheaper than people: is it worth the effort?

• How often and for how long will the code be run?
• How many hours, days or weeks should be spent?
• Find the sweet spot:
  • Compiler and/or library optimisations quick;
  • code tweaking can take months. . .