In Australian English, with reference to the supposed destruction of the brain by white ants (i.e., termites), the plural noun white ants is used of loss of sanity, sense or intelligence.

(The singular noun white ant occasionally occurred in early use.)

—Cf. also a sandwich short of a picnic and not to have both oars in the water.

These are, in chronological order, the earliest occurrences of white ant(s) that I have found:

1-: From the fourteenth chapter of Thus He Was Saved, by Horace Earle, published in The Week: A Journal of Commerce, Farming, Mining, & General Information & Amusement (Brisbane, Queensland) of Saturday 31^st May 1879:

“Now what are you going to do?”
“I don’t know any more than that we are going abroad soon,” replied Sandy.
“To Queensland?”
“To Australia somewhere,” said Sandy, “I can’t say exactly where.”
[…]
“I am pretty familiar with the dodges of that country, and can give you a few valuable wrinkles. A new chum has a lot to put up with, and for a time is very green, very green indeed. In many cases before he cuts his eye teeth he gets the white ant in him, and loses heart.”
“What do you call the white ant, Tom?” inquired Sandy curiously.
“Well, the white ants are little insects that play the very mischief with everything they come across, and will eat the very stretcher from under you, if you don’t take care.”
“But you don’t mean to tell me that they get inside a man?”
“It doesn’t matter what it is to them; but I need the—the—what do you call it in a—a—a—. I must jerk it up, Sandy, the proper words never come to mo when I want them. What I mean is that unless a new chum has got no end of pluck, he gets sick and sorry, and wishes he was home again.”

2-: From a letter to the electors of Fassifern, by ‘One of Yourselves’, published in The Queensland Times, Ipswich Herald and General Advertiser (Ipswich, Queensland) of Tuesday 8^th May 1888:

Fellow-electors, you must not shilly-shally with this coolie question, nor let these followers of Mr. Salkeld stuff your heads with arrant nonsense about wanting coolies to mow your hay unless you get protection. Let old Dry-as-Dust know that our fellow-elector, Mr. Bullmore, is our safeguard, and that he will not have coolie immigration as an alternative measure in case of a protection measure failing. Let old Dry-as-Dust know that Mr. Bullmore has taken the bull by the horns, and will steadfastly oppose coolies, no matter what other measure may be carried or what other measure may be overthrown.
[…] It has just dawned upon me now that this old dry-as-dust man must have the white ants very bad.

3-: From a letter to the Editor, published in The Northern Miner (Charters Towers, Queensland) of Friday 2^nd January 1891:

I doubt that the supporters of the scheme of Italian cheap labor will say that the writer of the above has not given much of his time to the study of political economy, and that he has got the white ants bad, and a fit person to be sent to Woogaroo Asylum for lunatics.

4-: From the account of a Labor meeting, published in The Daily Northern Argus (Rockhampton, Queensland) of Tuesday 5^th March 1895:

The speaker went on to refer at length to the attitude of the Labor Party in the Legislative Assembly, being now and again interrupted by a stentorian voice asking
          An Utterly Irrelevant Question,
“What about the £15,000 voted to Sir William Macgregor for New Guinea?” The interruption was so malapropos that everyone had to smile. Barcoo’s representatives suggested that the man had got white ants, to which the speaker acquiesced “Yes, he’s got them bad,” and proceeded.

5-: From Australian Slang, published in The Evening Standard (London, England) of Friday 7^th June 1895:

“Off his pannikin,” “A shingle short,” “To have white ants” (the last a Queensland phrase), are expressions for “wrong in the head.”

6-: From Truth (Sydney, New South Wales) of Sunday 10^th April 1898—the noun white ants is part of an extended metaphor:

The Reidrade dailies are always enthusing over the number of French and German vessels that come cargo-carrying to Sydney and back. Perhaps their white elephant brains have the white ants of the wooden head, perhaps—well perhaps they have no brains to dry rot, but they distinctly fail to grip the position. The foreign ships they are so fond of, especially the German, are highly subsidised by their Governments, and so are enabled to offer a low freight rate, whereby they do immense injury to British ship-owners. It is indeed curious to see freetrade editors absolutely advocating on the part of Germany what amounts to a very stringent form of protection.

7-: From Answers to Correspondents, by ‘Gossip’, published in The Sydney Stock and Station Journal (Sydney, New South Wales) of Friday 17^th June 1898:

Some people want to kick me for what I say. If some of them knew how much I didnt [sic] say, they’d bless me for a philanthropist! I’d give a £5 note to the Fresh Air League for the privilege of saying what I thought about some things. When a man gives me credit for “speaking out” it makes me blush, when I remember how much I don’t say. But I’d better shut up eh? I haven’t got “white ants” or anything of the sort, but I’m glad I found you out.

8-: From The Geraldton-Murchison Telegraph (Geraldton, Western Australia) of Thursday 23^rd June 1898:

“White Ants.”
By W. M. C., in the Australasian.

“White ants” in Far West Australia is the familiar term applied when a bushman or miner has “got ’em bad.” That, is, when, almost driven to despair by the desolateness of his surroundings in the far interior, he over-indulges. During my three-years’ experience in official and journalistic capacities, especially the former, many instances of “white ants” came under my notice. One in particular may be worthy of recounting. After a hard spell of presswork, I accepted a Government situation which required that I should travel far out into the unknown. I was assured it was in a grand district, 600 miles N.E. from Perth, with mountains, lakes, and woods—away out on the frontier near the unexplored desert, and bounded by nothing in particular. It was a field with an area about equal to the whole of Victoria. After selecting the site of the future capital, I had to instal myself as a sort of Pooh-Bah, being the only official within the limits.
[…]
It was a gray, sultry sort of place out back in the rainless desert. Mines were being developed, and when wages were paid once a fortnight, out of English capital, “the field” mounted on bikes, brumbies, and camels, and had two days sport.
It was on a Sunday when the case of “white ants” forced itself on my attention. I had spent a long night writing official letters, to be forwarded by cyclist, at 1/ per letter, to the nearest post-office, 100 miles away, and I was indulging in a morning’s rest.
“Good morning, sir, are you awake?” brought me to a sitting posture.
“What can I do for you?” I inquired.
“I have come to be hanged. Yes, come to be hanged. I killed them all! Father, mother, and Dick! Now, Jack says I must not further disgrace the family, and rather than have me hanged he will bury me up to my neck in sand, and let the crows pick my head off. But—I don’t want that, and he is after me.”
I understood the situation, and said, soothing and cheerfully, “Oh, I remember I tried your case yesterday, and let you off with a caution, but you know, Charlie, you must not do it again.”
“Oh, thanks very much. Have you seen them?” “Seen who?” I asked, as I rose and walked to the tent door in my pyjamas. Looking along the track I saw some men near the hotel, and said, “Oh, yes, there they are.”
He looked, saying, “Then, I must be going,” darted into the mulga before I could lay a hand on him.
Two days passed, and I had just turned in when a voice asked “Have you gone to roost?”
“Yes, but come in all the same,” I replied, and a bronzed miner entered.
“Awfully sorry to disturb you, but I have come to report that my mate Charlie, who was having a bit of a spree and got ‘white ants’ has been missing since Saturday last. Can’t find him anywhere. What is to be done?”
“It must have been Charlie who came here on Sunday to get hanged,” I said.
Then I proposed to borrow the services of Mr. Blank’s black-tracker.
The tracker came next morning, and got all particulars. In the afternoon he returned, and what a scarecrow came with him. Three day’s exposure to the blistering sun, without food or water, had wrought terrible havoc on Charlie. He was only partly dressed. His lips were black and cracked; his head was smeared with blood, which still oozed from two wounds—one in front and the other at the back, near the base of the skull. His eyes showed no more intelligence than two burnt holes in a blanket. The white ants had had a terrible innings. I had him placed in a covered-in space between my two tents, and tied him by the feet, as he was restless and raving, and could not be trusted to lie on a stretcher. A pint of hot Bovril soothed him, but towards midnight he became violent, protesting against being buried alive in the sand. What was to be done? My only drug was a phial of homœopathic belladonna. Should I try its effects? Well, if he died I should have to be coroner, and would certainly not return a verdict of wilful murder against myself. So I dosed him, washed the blood from the holes in his head, and covered them with kerosene rags to keep the flies off. He slept until noon next day—slept until I grew afraid—but he wakened up refreshed and more like a man. I poured the extract of meat into him, and watched his recovery. Late at night he begged for “more of that stuff” (belladonna), and I complied by giving him small doses. Within a week I had the satisfaction of seeing him move about clothed in his right mind.
When questioned about the holes in his head, he had a confused recollection of trying to beat his brains out with two large pieces of quartz to escape being buried alive.
Fearing that he might again give way to drink, I cautioned the firewater vendor, and hinted at enforcing all the penalties and pains if he supplied a single drop of forked lightning to my first patient. I also gravely told Charlie that I had put him under the “Dog Act” for six months—that is, under prohibition for his own good. He left distressed and downcast.
Weeks passed, new publichouses and other places sprung up in that wilderness like Jonah’s gourd. One blistering day Charlie came to my quarters, looking well, but downcast.
“How are you jigging, old saw?” I inquired heartily.
“Well, but disgraceful,” he replied. “Oh, can you remove the ban—take away the curse of the Dog Act? It would break my poor mother’s heart if she knew of my disgrace.”
“Do you feel strong enough to resist temptation?” I said kindly.
“I do, and promise on my honour never to indulge again—never!”
“Well, my boy, you have a clean sheet so far as I am concerned. I only cautioned vendors, but never entered your name on the blacklist.”
“This is too much. I cannot thank you sufficiently for saving my life and my honour. Come down to the pub., and see how I can withstand it.”
I went reluctantly. Charlie had bitters and ginger-beer only. I saw him frequently during many months after that, and often spent an hour with him, but on no occasion did I ever see him take anything stronger than an unsmokable cigar made out of cabbage-leaves of the kind sold at a shilling a bundle in civilised parts, but retailed as a favour on the fields at a shilling each.