The literal meaning of the American-English phrase to take the Fifth Amendment is: to appeal to Article V of the ten original amendments (1791) to the Constitution of the United States, which states that:
No person […] shall be compelled in any criminal case to be a witness against himself.
The earliest instance that I have found is from an article titled Patriotic Communist?, published in the New Castle News (New Castle, Pennsylvania) of Tuesday 9th March 1954; reporting from Washington D.C., Frederick C. Othman explained that Senator Joseph McCarthy (1908-1957) identified Professor Frank McGee, who was appearing before him,
as the one-time educational leader of the Communist cell of the Federal telecommunications laboratories, doing secret radar work for the Army at Nutley, N.J. Prof. McGee, who said he used to teach at the University of Kentucky, wouldn’t talk about that Communist cell business.
He sought refuge under the Fifth amendment to every question involving his alleged membership in the party. “But every time I take the Fifth amendment,” he added, you conclude the answer is yes, anyhow.”
(Incidentally, there is an error in the Oxford English Dictionary (2nd edition, 1989): the earliest instance of to take the Fifth Amendment is stated as being from The Robesonian (Lumberton, North Carolina) of 19th March 1953, but the sentence that this dictionary quotes (“He said he would take the Fifth Amendment today.”) appeared in The Robesonian on 19th March 1958.)
The earliest instance of the elliptical form to take the Fifth that I have found is from the Chicago Sunday Tribune (Chicago, Illinois) of 2nd December 1956:
TELLS HAWAII’S RED DELEGATES TO DEMOCRATS
Ex-Governor Says 15 ‘Took the Fifth’
By Wayne Thomis
(Chicago Tribune Press Service)
Honolulu, Dec. 1—Communists have infiltrated Hawaii’s Democratic party to such an extent that in 1952 15 delegates from the islands to the Democratic national convention were men who had “taken the fifth amendment” when asked whether they were members of Red cells.
Ingram M. Stainback, World War II governor of Hawaii, now a territorial supreme court justice, told this to members of the United States Senate internal security subcommittee on the second day of hearings here. He said “taking the fifth” is communist slang for hiding behind the constitutional privilege which exempts a man from forcibly incriminating himself.
The phrase to take the Fifth (Amendment) has come to be used generically to mean to decline to reveal confidential matter about oneself. The earliest instance of this generic use — with wordplay — that I have found is from Glamorous Star Just Loves Kids, by Bill Roth, published in the Sunday Call-Chronicle (Allentown, Pennsylvania) on 24th August 1958:
There was this Hollywood star making a personal appearance.
And I ups to her suite of rooms for an interview and right away I knew I was in Dutch.
I shudda known better than ask her was she a natural blonde.
“Never in my life have I hit the bottle — peroxide, that is — but it’s guys like you with their silly questions that almost make me want to take the Fifth — gin, that is. That and these three-a-day appearances — they’re getting me down.
“Imagine, I just got in town and I’m whisked right out to Dorney Park for these personal appearances — 4-7-9, 4-7-9. Gee it’s getting monotonous.
“If it weren’t for these wonderful kids, I’d give it all up.”
I just tickled her under the chin a bit and said “C’mon girl, buck up.”
And I knew from the way she laid her head on my shoulder that all was forgiven.
Yes, boys and girls, it’s Lassie we’re talking about, in town for personal appearances at Dorney Park.
PERSONAL APPEARANCE—“I just love Allentown,” Lassie is here telling a Call-Chronicle reporter as she takes time out from a three-day routine at Dorney Park. She’ll soon return to Hollywood to begin a series of TV dramas.
from the Sunday Call-Chronicle (Allentown, Pennsylvania) — 24th August 1958