“I think we may say everything’s more or less oojah-cum-spiff. With one exception, Jeeves,” I said, a graver note coming into my voice as I gave Gus his second helping of kipper. “There remains a fly in the ointment, a familiar saying meaning—well, I don’t quite know what it does mean. It seems to imply a state of affairs at which one is supposed to look askance, but why, I ask myself, shouldn’t flies be in ointment? What harm do they do? And who wants ointment anyway?”

Much Obliged, Jeeves (1971), by the English author Pelham Grenville Wodehouse (1881-1975)

The phrase a fly in the ointment means a minor irritation that spoils the success or enjoyment of something.

It is an allusion to the first verse of the Book of Ecclesiastes, 10:

King James Version (1611):
Dead flies cause the oyntment of the Apothecary to send forth a stinking sauour: so doeth a little folly, him that is in reputation for wisedome and honour.

The word ointment denotes a smooth oily substance, often mixed with some medicament, which is rubbed on the skin for medicinal purposes, or as a cosmetic.

In the biblical verse, the word refers to a cosmetic. In the Literal Translation of the Bible (an extremely literal translation that attempts to preserve the tense and word usage as found in the original Greek and Hebrew writings), by Robert Young (1822-88), the verse is:

(1863 edition)
Dead flies cause the perfumer’s perfume
To send forth a stink;
The precious by reason of wisdom
From honour—a little folly!

And in the New International Version, the verse is:

As dead flies give perfume a bad smell, so a little folly outweighs wisdom and honour.

The earliest uses of a fly in the ointment that I could find are from A Body of Practical Divinity, published in 1692, composed of sermons by Thomas Watson (1620-86), English non-conformist Puritan preacher and author:

What is Iustification?
Resp. It is an Act of God’s Free-grace, whereby he pardoneth all our sins, and accepts us as righteous in his sight, only for the Righteousness of Christ imputed to us, and received by Faith.
Iustification is the very Hinge and Pillar of Christianity; and an Errour about Iustification is dangerous, like a Crack in the Foundation, or an Errour in the first Concoction. Iustification by Christ is a Spring of the Water of Life, and to have the Poison of corrupt Doctrine cast into this Spring, is damnable. It was a Saying of Luther, That after his Death the Doctrine of Iustification would be corrupted. As it hath been in these latter Times, the Arminians and Socinians have cast a dead Fly into this Box of precious Oyntment.
[…]
To do Gods will sincerely, is to do it with a pure Eye to Gods Glory. The Pharisees did the will of God in giving Alms, but that which was a dead Fly in the Ointment, was, that they did not aim at Gods Glory, but vain-glory.
[…]
Mat. 6.5. “Be not as the Hypocrites, for they love to pray standing in the corners of the streets, that they may be seen of men.” Prayer is good, but to pray to be seen of men, this was a dead fly in the Box of Oyntment.