It is a shortening of the more explicit no more chance than a cat in hell without claws. The earliest instance of this phrase that I have found is from Jackson’s Oxford Journal of 29th September 1753:
Poor John Billingsgate the late Object of Publick Justice, is now a deplorable Spectacle, which by the singular and uncommon Circumstances attending his Condition, draws the Attention of all the Civilians, Cockfighters, and Anatomists of this Place. Since his Tongue was cut out he has had no rest Day or Night […]. When his Phrenzy is highest, he makes Signs for Pen, Ink and Paper; but these are now denied him, for the Experiment was made whether Writing would give him any Relief; and it was found that it was only adding Fuel to the Flame. The only intelligible Sentence he has wrote, is the following: “Without a Tongue I have no more chance in Life, than a Cat in Hell without Claws.” Then he grew quite delirious.
The English antiquary and lexicographer Francis Grose (1731-91) wrote, in A Classical Dictionary of the Vulgar Tongue (2nd edition – 1788):
Cat’s Foot. To live under the cat’s foot; to be under the dominion of a wife, hen-pecked. To live like dog and cat; spoken of married persons who live unhappily together. As many lives as a cat; cats, according to vulgar naturalists, have nine lives, that is, one less than a woman. No more chance than a cat in hell without claws; said of one who enters into a dispute or quarrel with one greatly above his match.
The Gentleman’s Magazine: and Historical Chronicle (London) of April 1792 published a humorous poem titled A Squib, From Dr. Jortin’s Tabby-Cat, Addressed to him from the shades below, which thus begins:
By kicks and cuffs worn out, I seek below
That shore, where many a two-legg’d cat will go*.
* [A footnote suggests this humorous explanation of the phrase:] Charon has orders to deprive all cats of their claws. Whence comes that saying in hopeless cases—“he has as much chance as a cat in Hell without claws?”
(In Greek mythology, Charon was an old man who ferried the souls of the dead across the Rivers Styx and Acheron to Hades.)
The British politician and pamphleteer Sir Philip Francis (1740-1818) did not use without claws in the following letter:
London, May 5, 1769.
My dear Brother,—[…] We have politics enough, God knows, but as I have not the honour to be entrusted with the secrets of either party, I can give you nothing but what you will see much more elegantly set forth in the newspapers. Truth is out of the question. Each party says and believes just what suits themselves without decency or moderation, and a neutral party is detested by both. A philosopher has no more chance among them than a cat in hell. (I wish, by the by, that the person who stole my cat were in the warmest corner the devil could find for him.)
The earliest instance of not a cat in hell’s chance that I have found is from the Yorkshire Telegraph and Star of 11th September 1914, in which Private J. Brown, of the King’s Own Scottish Borderers, explained how a British infantry brigade “held at bay the Germans in face of fearful odds” at St. Ghislain in Belgium during World War One:
“To use a soldiers’ phrase our men if they had advanced in the open would not have had a ‘Cat in hell’s chance.’”