The phrase bread and butter is used in negative and interrogative constructions—especially it’s no bread and butter of mine (also of yours, etc.)—to denote a matter which one has the right to express an opinion on, involve oneself in, etc.

This phrase occurs, for example, in The Marquis She’s Been Waiting For (New York: Lyrical Press, Kensington Publishing Corp., 2019), by the U.S. novelist Ella Quinn:

During supper of weak tea, stale bread and butter, and dry cake, she entertained him with stories of her brothers and sisters. The few times she did mention Dursley, Mrs. Chatham pursed her lips together in disapproval. Was there some bad blood between the families? He hoped this was not a Romeo and Juliet situation. Yet if that was the case, why would Dursley’s sister be assisting her? Not that it was any bread and butter of his. He would assist the lady in any way he could to keep her out of Lytton’s clutches. Sort of marrying her that was.

These are, in chronological order, the earliest occurrences of the phrase that I have found:

1-: From The History of the most Renowned Don Quixote of Mancha: And his Trusty Squire Sancho Pancha (London: Printed by Thomas Hodgkin, 1687), the translation by John Phillips (1631-1706?) of El ingenioso hidalgo (in Part 2, caballero) don Quijote de la Mancha (1605-15), by the Spanish novelist and playwright Miguel de Cervantes Saavedra (1547-1616). It is Sancho Panza who utters the phrase on several occasions, in dialogues with Don Quixote—for example:

Sancho, said he, what shall I do? I have laid violent Hands upon a Clergy-man, and I am afraid of being Excommunicated; according to that same Decree i’ the Spiritual Court, If any one, through the Temptation of the Devil, &c. Yet now I think on’t, I never touch’d him with my Hands, but only with my Lance. Besides, I do not believe they were Priests, or Men that any way belong’d to the Church, but meer Hobgoblings and Ghosts. That’s no Bread and Butter o’ mine, reply’d Sancho, I’m sure I struck no body, nor no body struck me, which makes me believe they were Priests, and not Ghosts nor Apparitions.
[…]
Sancho, reply’d Don Quixote, […] she stood in need of the wholesom Counsels of Mr. Tom Thumb; which gave Occasion to that false and scandalous Report, that they lay together. But I say again, they lye, and lye a thousand times over, whoever they be, that shall either report, or so much as think such a Calumny against the Queen of Trumps.
Why, Sir, quo Sancho, I neither say, nor think one way or t’ other; let them that affirm it eat the Lye, and swallow it when they ha’ done. If they lay together, th’ have answer’d for it before now. I come from plough, I know nothing; I never thrust my Nose into other mens Porridge; my Name’s Twyford, I neither meddle nor make. He that buys and sells, shall find his Gains in his Purse. Naked I came into the World, and naked I must go out. I neither take, nor give; I neither lose, nor gain: if they lay together ’tis no Bread and Butter o’ mine. Who can make a Silk Purse of a Sow’s Ear? But if ye catch me i’ your Corn, put me i’ your Pound.

2-: From a letter, dated 12^th August 1732, by the Irish satirist, poet and Anglican cleric Jonathan Swift (1667–1745)—as published in The Works of Alexander Pope (London: Printed for A. Millar; J. and R. Tonson; H. Lintot; and C. Bathurst – 1757):

Mr. Gay is not discreet enough to live alone, but he is too discreet to live alone; and yet (unless you mend him) he will live alone even in your Grace’s company. Your quarrelling with each other upon the subject of bread and butter, is the most usual thing in the world; Parliaments, Courts, Cities, and Kingdoms quarrel for no other cause; from hence, and from hence only arise all the quarrels between Whig and Tory; between those who are in the Ministry, and those who are out; between all pretenders to employment in the Church, the Law, and the Army: even the common proverb teaches you this, when we say, It is none of my bread and butter, meaning it is no business of mine. Therefore I despair of any reconcilement between you till the affair of bread and butter be adjusted, wherein I would gladly be a mediator.

3-: From A New Plain and Useful Introduction to the Italian. Compiled from the best Grammarians, who have wrote in the Tuscan Language. Together with a Choice Collection of Italian Idioms. Collected from the most noted Authors, with the proper English adapted: For the Benefit of such as are desirous to have a thorough Knowledge of, and Speak correctly, this useful and beautiful Tongue (London: Printed for J. Wilcox, 1739), by John Kelly (1680?-1751):

Non e il tuo caso. We say, ’tis no bread and butter of yours; ’tis none of your business.

4-: From The Magician, or the Bottle-Conjurer, published in The Life and Uncommon Adventures of Capt. Dudley Bradstreet (London: Printed and Sold by S. Powell, 1755), by the Irish adventurer, government spy and confidence trickster Dudley Bradstreet (1711-1763):

Set. Why do you provoke our Master thus, Grip?
Grip. Setter, what have you to do with it; is it any Bread and Butter of yours?

5-: From The Stage-Coach: Containing the Character of Mr. Manly, and the History of His Fellow Travellers (Dublin: Printed by Henry Saunders, 1762), by Miss Smythies:

“He told me as how a friend of his was going to be married to a young gentlewoman, and her friends would not agree to it, though she loved him as her life; whereupon, and please your worship, he said she had agreed to tell her friends that she must come to my house to meet one Mrs. Adams, and so they proposed to go to the Fleet and get married. This was the truth of the matter for certain, or I would not have meddled or made with it, for it was no bread and butter of mine you know, though the gentleman behaved himself like a gentleman, that I must say, and paid nobly for what he had; yet if so be I had known he had designed to deal unhandsomely by the gentlewoman, in any shape in life, he should not have brought her here.”

6-: From Another Traveller! Or Cursory Remarks and Critical Observations made upon a Journey through Part of the Netherlands in the latter End of the Year 1766 (London: Printed for Johnson and Payne; T. Cadell; and J. Robson – 1769), by Samuel Paterson (1728-1802):

A Dutchman is always wrapt up in himself, whatever chances to be his condition.
He is smoking his pipe—and you disturb him:—He is meditating upon his own business—and you interrupt him.—’Tis true, you hired his chaise at a certain rate, to transport you from this place to that—which he will faithfully perform in the usual time—there ends your contract:—but you did not hire him to be your gazetteer and interpreter.
Idle curiosity is sure to be baffled by such fellows.—He will either be deaf to the question, or surly, if repeated, or ignorant touching the matter questioned, or unsatisfactory in his answer.
[…]
Shall we be there by dinner-time, think you?—‘Ik verstaa u niet! I don’t know what you mean.’—What fine castle is that?—‘’T gaat my niet aan!—that’s no bread and butter of mine,’ says the Dutchman.

7-: From The Talkative Woman, by John Buncle, Junior, published in Drewry’s Derby Mercury (Derby, Derbyshire, England) of 26^th July 1776:

“I believe nobody is duberous whose Child it is.”
“What, do they suspect ††††?” says Charles, supplying the Want of a Name with a significant Wink, and placing his Fore-finger along the Side of his Nose.
“Well, well, it’s no Bread and Butter of mine, G— knows.”

8-: From the second edition of A Classical Dictionary of the Vulgar Tongue (London: Printed for S. Hooper, 1788), the English antiquary and lexicographer Francis Grose (1731-1791) recorded the following phrases:

Bread and Butter Fashion. One slice upon the other. John and his maid were caught lying bread and butter fashion.—To quarrel with one’s bread and butter; to act contrary to one’s interest. To know on which side one’s bread is buttered; to know one’s interest, or what is best for one. It is no bread and butter of mine; I have no business with it.