The phrase to have a tiger by the tail, and its variants, mean: to find oneself in a situation that has turned out to be difficult to control but cannot be got out of.

The image is that someone holding a tiger by the tail can neither keep hold of it nor let go of it with safety.

This phrase occurs, for example, in the following book review, by Felicity Newson, published in the Liverpool Echo (Liverpool, Merseyside, England) of Saturday 11^th July 1998:

A State Of Shock by Nicholas Salaman (Harper Collins, £5.99)
THIS is a courtroom drama with some sinister topspin set against a brooding backdrop which presents Britain as a threatening place in which to live.
The ‘wig’ is Heather Semple, a young barrister with attitude and ambition. Keen to acquire more clients and a higher profile, she agrees to defend a man accused of a very vicious rape and murder.
But Heather rapidly discovers that she has a tiger by its tail. As her encounters with the defendant develop, she is alarmed at how quickly she has been drawn into his world.
The panic is justified. Heather is quickly thrown into a terrifying vortex of evil and corruption, which Nicholas Salaman puts across well.

The earliest occurrences of the phrase to have a tiger by the tail and variants that I have found are as follows, in chronological order:

1-: From the deposition of one Jeronimo de Cruz, in Depositions of European and other captives imprisoned by the Burmese Government during the late war (depositions taken in May 1826), published in the Appendix to Journal of an Embassy from the Governor-General of India to the Court of Ava, in the Year 1827 (London: Henry Colburn, 1829), by the British diplomat John Crawfurd (1783-1868):

Q. Did you ever hear the Burmans express regret for having entered into a war with the English?—A. Yes, very often. A person from the palace told me, that three months after the arrival of the English at Rangoon, he heard the King say, “He was in the predicament of a man who had got hold of a tiger by the tail, which it was neither safe to hold nor let go.”

2-: From a letter to the Editor, published in The Craftsman (Rochester, New York, USA) of Tuesday 10^th March 1829—however, here, the phrase to take the tiger by the tail seems to mean to attack a difficult problem fearlessly, to ‘take the bull by the horns’:

New York, Feb. 24, 1829.
Dear Sir,—
Your second number of the “Craftsman” has just been received. I read the first number with much attention and much pleasure, and would have immediately acknowledged to you the high opinion I entertain of its character and principles, but I purposely postponed it till I should read the next number.
To say that your paper was handsomely got up, and ably executed, would be but common praise. I can now say more: I believe it promises to be one of the best weekly journals of its kind in our country, and if you continue to conduct it with the same spirit and independence which your first two numbers evince you capable of, a generous and enlightened publick will most assuredly reward your exertions.
You have literally taken the tiger by the tail in his own native woods, and although he is of the feline species, and like the cat, may have nine lives, the weapons you employ will eventually destroy the monster and preserve the innocent and unwary from his prowling grasp. Ignorance in the great body of mankind of their religious and political rights, induces the more knowing and crafty of our species to lord and tyrannize it over their less gifted brethren, and throw every impediment in the way of self judgement and self control. Educated from infancy to believe that royalty is from heaven, the vassals of European governments, however wretched and miserable their condition, and however borne down by oppression, will still sacrifice their last blood in support and defence of the very governours and institutions that afflict them. They act and believe as they have been taught. To touch upon early instilled prejudices, to develope [sic] human reason and direct its course in the path of common sense, to break down the barriers of superstition and prejudice, those two famous pillars of our early education, and to teach man his own powers and consequence, is a task that none of our publick journalists have dared to undertake. You sir, have avowed the courage to do that in which every patriotick, virtuous and honourable mind will support you. You must expect to meet with difficulties and dangers—so did our fathers in the revolution. As against them, there will be against you, a well disciplined army of zealots, who will dispute every inch of reason and common sense—you will meet with “hard knocks,” and a thousand privations, but I trust your spirit of independence and your native propensity to perseverance, will ultimately gain you the laurel of victory.

3-: From the State Gazette (Trenton, New Jersey, USA) of Tuesday 18^th November 1851:

In a Bad Fix.—Austria wishes to reduce her army, but she cannot do it. She dare not dismiss a single regiment, while she maintains her existing tyranny; and, at the same time, financial ruin must rapidly bring on a convulsion, unless such reduction is resorted to. The same condition of things prevails in France; and in both cases it is so palpable as to induce the violent among the Democrats to forbear urging the outbreak, which is the only thing that could retard or avert the approching [sic] end. These governments are in the condition of the man who had the tiger by the tail; he could neither hold on nor let go with safety.

4-: From one of the unconnected paragraphs making up the column A Little of Everything, published in The Washington Examiner (Washington, Pennsylvania, USA) of Saturday 13^th December 1851:

—If a man has a tiger by the tail, which would be best for his personal safety—to hold on or let go.