The British- and Irish-English phrase (fingernails) in mourning for the cat means dirty fingernails.

I have not found out why for the cat is used, but what I have noted is that this phrase is an extension of the earlier nails, or fingernails, in mourning, in which the dirt edging the fingernails is compared to the black border edging mourning paper—as explained in The Children’s Corner. Conducted by Uncle George, published in The Clifton and Redland Free Press (Bristol, Bristol, England) of Friday 11^th June 1909:

My dear Boys and Girls,
[…]
We were talking about the hand last week. […] It may be well to request you to look at it and enquire if it is clean and whether your nails are in mourning?
“Nails in mourning, Uncle George? What ever do you mean?”
Nails are something like black-edged note paper and have a border of dirt under them sometimes.

The earliest occurrences of the original phrase that I have found are from the translation(s) into English of the memoirs of Marie-Fortunée Lafarge (née Capelle – 1816-1852), a Frenchwoman who was convicted of murdering her husband by arsenic poisoning in 1840:

1-: From Memoirs of Madame Laffarge [sic], written by herself, published in The Athenæum: Journal of English and Foreign Literature, Science, and the Fine Arts (London, England) of Saturday 28^th August 1841:

Clementine had taken in hand the reformation of M. Laffarge’s dress; knowing my taste, she told him what colours I preferred; made him put on the cravat which would please me; prohibited the waistcoat with glaring colours, which would have appeared to me bad taste. Upon her advice, M. Laffarge shaved every day, paid attention to his hair, his shoes; put on large gloves when he went to the forge, and spared me two calamities, insupportable in private life,—slipshod shoes, and nails in mourning, which are, I think, infallible preservatives against love.

2-: From the Cheltenham Chronicle, and Gloucestershire Advertiser (Cheltenham, Gloucestershire) of Thursday 30^th September 1841:

(From our own Correspondent.)

Paris, September 26, 1841.
[…]
Madame Laffarge [sic] has published her memoirs in three volumes. […] She thus describes her first interview with M. Laffarge:—“Imagine, my dear, a stout heavy man, with feet an ell long, coarse hands, nails in mourning, a red cravat, a shirt-collar cutting his ears, an embroidered waistcoat, boots not varnished, gloves not lemon colour, …… no gloves at all!—thick lips, moist with saliva, …… a man with a sourish smell.”

The earliest occurrence of the phrase (fingernails) in mourning for the cat that I have found is from the account of a demonstration against the Education Bill organised by the Lincoln Diocesan Board of Education at Lincoln on Saturday 27^th October 1906—account published in the Sheffield Daily Telegraph (Sheffield, Yorkshire, England) of Monday 29^th October 1906; among the orators was a “typical Cockney […] with his torrential eloquence and cock-sure cockneyisms” named Mr. J. C. Collings:

In Mourning the Cat.

“Don’t be too squeamish about the methods you adopt in fighting your enemies,” was a bit of advice he illustrated by one of his numerous cockney stories.
A little boy in one of the Whitechapel schools appeared in class every morning with his “fingernails in mourning for the cat.” The teacher punished without result.
“Why not try kindness?” suggested Mr. Collings.
Next day the teacher asked: “Now what should you say if I came to school with my fingernails in mourning for the cat?”
“Please, mum,” replied the sharp little urchin, without a moment’s hesitation, “I should not say anything. I’m too perlite.”
Mr. Collings objects to Churchmen being too “perlite” in their opposition to the Bill.

The second-earliest occurrence of the phrase that I have found is from Paddy Clarke Ha Ha Ha (London: Secker & Warburg, 1993), a novel by the Irish author Roddy Doyle (born 1958):

My ma once smelt the smoke off me. She saw my hands first. She grabbed one of them.
—Look at your hands, she said.—Your fingernails! My God, Patrick, you must be in mourning for the cat.

Finally, the following is from A Dictionary of Catch Phrases British and American, from the Sixteenth Century to the Present Day (Taylor & Francis, 2005), by Eric Partridge and Paul Beale:

you’re in mourning for the cat. Your finger-nails are filthy: proletarian: C20; by 1970, somewhat outdated. P.B. [= Paul Beale]: in later C20, still occ. applied to someone whose trousers are too short to cover his ankles.