With reference to a dog used for hunting game, the colloquial American-English phrase that dog won’t hunt, and its variants, are used to express the opinion that a particular plan or approach will not succeed.
—Synonym: that cock won’t fight.

I have found an isolated 19^th-century occurrence of the phrase that dog won’t hunt in the following from The Globe (Washington, D.C.) of Tuesday 15^th August 1843:

INDIANA.

The returns which we published yesterday, and the letter from Indianapolis (the seat of Government of Indiana) and the “extra” from the State Sentinel (Chapman’s paper) which we now publish below, “tell the tale” for this State. The Democrats have without doubt, we think, elected their candidate for Governor; a majority of members of Congress; and a majority to the State Legislature. The returns which we have received from four of the six counties in the 3d congressional district, and the remark which our correspondent makes relative to Rush county, (which is in it,) lead us to believe that the Hon. Thomas Smith, “the gentleman tanner,” as he is called, is certainly elected. We know him well: he would be an honor to any profession. The district gave 1,821 Whig majority in 1840.
We think our correspondent errs when he says, “But that dog won’t hunt.” It is our opinion that he hunts too much; runs after small game; “don’t stand at the tree;” “barks up the wrong tree;” is fit for nothing but a coon dog, and hardly that; and that, in the big hunt of ’44, he will be killed so dead, he will not be able to give one dying yelp.
Extract from a letter headed “IMPORTANT AND TRUE,” and dated
Indianapolis, Indiana, Aug. 9, 1843.
Gentlemen: I see, from an article in the Globe, that you do not hear from the Hoosier State very often; and so I have concluded to give you a little news.
[…]
Don’t let the friends of Mr. Calhoun and those of Mr. Van Buren get at loggerheads, or anything beyond amicable sparring. We are ready to go for either of them, or for any other sterling Democrat, provided always that he will fight rather than give up Oregon, or any other portion of our country. Cass is much a favorite in this State. I put in a word (formerly) once in a while for Captain Tyler ¹. “But that dog won’t hunt.”

¹ This refers to Robert Tyler (1816-1877), the eldest son of John Tyler (1790-1862). John Tyler was the tenth President of the United States (1841-45), and, in 1843, Robert Tyler sought in vain the presidential nomination.

The other early occurrences of the phrase that dog won’t hunt, and variants, that I have found, are from The Times ² (Tuttle, Oklahoma):

² Texts 1 and 3 from The Times (The Tuttle Times in 1907) contain numerous spelling and grammatical mistakes.

1-: Of Friday 6^th September 1907:

Some local democrats are endeavoring to circulate some caluminous reports about Gov. Frantz, but “that dog wont hunt,” for Haskell, with all of his dirty method of campaining has been unable to start anything to hurt Frantz, and reports put in circulation by such ametuers as Tuttle possesses can’t be forced down the throats of the people, so you had as well call off your dogs.

2-: Of Friday 20^th September 1912:

A letter from A. W. Maxwell of the Guthrie Star to us regarding the editorial notice we gave him in our last is a most amusing proposition, indeed. He says he is just “as good a Roosevelt supporter as we are.” Now if he is such a hot cookie as a Roosevelter, why in the name of common sense and honesty does he carry the names of Taft and Sherman at the head of his announcement column? Why don’t he display his colors in his editorial columns if he is a Progressive instead of holding onto the old, dead issue, the republican party? No, no, Maxwell, you can’t be for Roosevelt and carry Mr. Taft at the head of your paper and make the people believe it. The dog won’t hunt. The people are not fools. Your action belies your words.

3-: Of Friday 23^rd January 1914:

THE DOG WON’T HUNT

In discussing the new currency in Oklahoma City a few days ago Senator Owen said that the iron hand of the government was behind the new measure. Who is it that composes the government, or should do so, but the great common people, the public, thru its officials? Then if that be true, and it is and Mr Owen will not deny it, then why did he oppose and defeat an amendment to the new bill to let the public subscribe the capital of the new banks? Why did he say, if he was not at the time working in the interest of big bankers of which he ss one, that banks would have nothing to do with the banks if they could not control them? If a man controls the banks that control the finances of the nation, does he not control the finances of the nation and the destinies of the nation as well? Mr. Owen, you may be a smooth talker, and may make those don’t follow the current events believe your spiel to the people in Oklahoma City, but to one who reads and thinks, your logic is wrong because you premise is wrong. You cannot possibly harmonize your acts and speech in the senate on that amendment and speech in Oklahoma, and you know it, or you should know it. You might say that was a small matter. Perhaps it was small thing, but it is by the small things that men do that the world know their value by. And besides, Mr. Owen, if you should say it a small matter of an amendment, it must have meant a great deal to the banks and you, that bankers should control the regional banks instead of public control of them, or you would not have opposed it so strenuously. No, Mr. Owen, you may be musician enough to play it on others and a piano, but you can’t it on those who think. Your dog will not hunt.