The colloquial phrase that cock won’t fight is used to express the opinion that a particular plan or approach will not succeed.
—Synonym: that dog won’t hunt.

This phrase refers to cockfighting, i.e., a blood sport in which cocks, often fitted with metal spurs, are set to fight each other, frequently in a specially constructed cockpit, and usually with spectators betting on the outcome.

These are, in chronological order, the earliest occurrences of the phrase that cock won’t fight that I have found:

1-: From a letter to the Editor, by ‘Rusticus’, published in The Loiterer ¹ (Oxford, Oxfordshire, England) of Saturday 5^th September 1789:
—Context: The author of the letter purports to be a rich and unmarried middle-aged country gentleman; he has accepted a pressing invitation to spend a week or two at the house of a distant cousin, who has been a widower for thirty years and has two maiden daughters:

“I have now,” said [my Cousin], “only one wish remaining, which is to see my girls (he called them girls, Mr. Loiterer) well married before I die.—I might have got them great matches, to be sure; dozens of Lords have been refused—but titles are not what I want—If I knew of any worthy gentleman of a tolerable good Estate, perhaps a thousand a year or so, and contented to live in the country and enjoy domestic happiness, I would be proud to unite him to Louisa to-morrow. I can give her—but no matter for that, she is an excellent young woman, and a fortune in herself.”
You may be sure that this eloquent harangue was not lost upon me, I immediately began to smoke ² the old Gentleman. “No, (thought I) that cock won’t fight.”

¹ The Loiterer was a humorous weekly periodical founded and largely written by two brothers of Jane Austen (1775-1817), James Austen (1765-1819) and Henry Austen (1771-1850), while they were undergraduates at the University of Oxford.
² Here, the verb smoke means to suspect of scheming.

2-: From Rosina: A Novel (London: Printed for William Lane, 1793), by the English author Mary Susanna Pilkington (née Hopkins – 1761-1839):

Rosina […] acquainted Austin with a design she had formed of passing for his daughter, an honour that quite confounded and astonished him.
“But now, my dear Miss Rosina, (said he,) do you think that story will hang well together? or that people would be such fools for to believe that a poor working man like me could have such a fine young lady as you for a daughter? Sure folks need only hear you speak one word, if you was drest in never such poor duds, for to know well enough that you are not one of the common sort! No, no! take my word for it, that cock won’t fight.”

3-: From Reynard the Fox: A renowned Apologue of the Middle Age, reproduced in Rhyme (London: Longmans, 1814):

For twenty oaths I would not budge
One inch to Rome; and as to trudge
Alone to th’ Holy City’s site—
I rather think ‘that Cock won’t fight!’
Here I’ll remain: no place is better.

4-: From the following dialogue between Admiral Culpepper and Young Contract, in The Boarding-House, Or, Five Hours at Brighton; a Musical Farce, in Two Acts (London: Printed for C. Chapple, 1816), by the English architect, novelist and playwright Samuel Beazley (1786-1851):

Ad. Oh! there she is—come along, sweetheart.
Y. Con. Come, my old one, that cock won’t fight—she’s my property.
Ad. I boil like a cauldron—I could knock him down.
Y. Con. What’s that?—any fight, my old one—is that your fun—then I’ll sicken you. (Boxing attitude.) I’ll fib you, my blue skin.

5-: From Eccentricity: A Novel (Dublin: Printed at the Hibernia Press for J. Cumming, 1820), by the Irish author Louisa MacNally (née Edgeworth):

“As to the young lady’s age,” said Dr. Sydney, “to my knowledge she wants some months yet of being twenty-one […].”
“Twenty-one!” vociferated M‘Manus, “oh, no, Doctor, that cock won’t fight. I lay you a rump and dozen she’s on the wrong side of twenty-two: catch weazels asleep, or ould birds with chaff—I know a trick worth two of that. Twenty-one, indeed!—oh, no, you won’t do that upon me, Doctor.”