The phrase town and gown, also gown and town, was originally used of Oxford and Cambridge, England. In this phrase:
– the noun town designates the non-academic inhabitants of a university city;
– the noun gown designates the resident members of the university.

(Here, the noun gown refers to the distinctive costume of a member of a university.)

The phrase town and gown occurs, for example, in the following from the Tunbridge Wells Courier (Tunbridge Wells, Kent, England) of Friday 17^th December 1999—Tonbridge School is a public school (i.e., a private independent fee-paying secondary school):

Nostalgia
Greyhound racing at the Angel ground—that’s just not cricket

THE possibility that the former Angel cricket ground at Tonbridge might be used for greyhound racing became a near probability until it was described as “the greatest calamity that could befall Tonbridge” at a town’s meeting in May, 1945, a few days after the European war ended.
The Congregational Church minister, the Rev R. O. Skinner, organised a petition and hoped to get 5,000 people to sign.
County Council Alderman William Nottidge presided and described dog racing as “merely a medium for gambling and betting”.
Unusually, given the traditional division between Town and Gown, the headmaster of Tonbridge School, Mr E. A. Whitworth, joined the protest.

The earliest occurrences of the phrase town and gown that I have found are as follows, in chronological order:

1-: From On Academical Gallantry (dated Cambridge, 3^rd March 1750), by ‘The Female Student’, published in The Student, Or, The Oxford and Cambridge Monthly Miscellany (Oxford, Oxfordshire, England) in 1751:

Here, I think, I may properly introduce a very singular gallant, a sort of mungrel between town and gown, being (as it were) of an amphibious nature, and partaking of both: I mean a bibliopola, or (as the vulgar have it) a bookseller.

2-: From A Candid Remonstrance to the Vice-chancellor, and Members of the University; occasioned by a late Address to the Freemen and other Inhabitants of the City of Oxford. By a Citizen (London: Printed for J. Cooke, 1764):

By the Assistance of our most vigilant Town-Clerk, who from his present Connections could easily gain us Admittance, I saw no Obstacle which could prevent a Compromise, and therefore exerted all my Powers for reconciling and confirming a lasting Alliance between TOWN and GOWN.

3-: From Christian vigilance. Considered in a sermon, preached at the Baptist chapel, in Taunton, on the Lord’s Day, after the sudden removal of the learned and Reverend Robert Robinson ¹. By Joshua Toulmin, M.A. To which is added, some account of Mr. Robinson, and his writings (London: Printed for J. Johnson and Thomas Knott, 1790), by Joshua Toulmin ²—as quoted in The European Magazine, and London Review: Containing the Literature, History, Politics, Arts, Manners & Amusements of the Age (London, England) of July 1790:

At the age of twenty-three, in the year 1759, he [i.e., Robert Robinson] was invited to a congregation of Protestant-Dissenters in Cambridge […].
He had not been long settled at Cambridge, before his singular talents and excellent qualifications began to be much noticed; and, at the desire of the Town and Gown, he set up a Lord’s-day evening lecture, which was crowded. It was supposed that no less than one hundred and fifty or two hundred gowns-men generally attended.

¹ Robert Robinson (1735-1790) was an English Dissenter, Baptist minister, hymn-writer and scholar.
² Joshua Toulmin (1740-1815) was an English religious minister and theologian.

4-: From Sketch of the History of Dissenting Churches: Origin of the Dissenting Churches at Cambridge, &c., by Robert Robinson, published in The Baptist Annual Register, for 1798, 1799, 1800, and part of 1801. Including sketches of the state of religion among different denominations of good men at home and abroad (London, England) in 1801:

While Mr. Holcroft ³ was minister at Bassingbourne, he formed a congregational church, consisting of many people of other parishes as well as of his own, beside several of both town and gown from Cambridge.

³ Francis Holcroft (1629?-1693) was an English religious minister.

5-: From Historical Account of Protestant Dissenting Churches in Cambridgeshire, by Robert Robinson and Joshua Toulmin, published in the Supplement to Volume V (January to December, inclusive, 1810) of The Monthly Repository of Theology and General Literature (London, England):

While Mr. Holcroft was minister at Bassingbourn he formed a congregational church, consisting of a great many people of other parishes, as well as of his own, besides several of both Gown and Town from Cambridge.
[…]
Mr. Robinson had not been long settled at Cambridge before his singular talents and excellent qualifications as a preacher, began to be taken notice of; and at the desire of the gown and town, he set up a Lord’s-day evening lecture.

6-: From the Index to Walks in Oxford; comprising an original, historical, and descriptive account of the colleges, halls, and public buildings of the University (Oxford: Printed by W. Baxter for R. Pearson, 1817), by W. M. Wade:

Animosities between Gown and Town, xvi. ⁴

⁴ This refers to “the old animosity subsisting between Gownsmen and Townsmen”, page xvi of the Introduction to the book.

7-: From The Battle of Peas-Hill, a poem by a certain Mac Quyllyam, in The Manksman’s Budget, published in The Brighton Magazine (London, England) of June 1822—an introduction explains: “The following effusion was penned the day after the memorable 13^th of November, 1820, which must be a day of pleasant recollection to all Cantabs, as long as there shall be a snob or radical amongst them, or a fist to bate them with.”:

Then rushed, undaunted, to the fight
The tall—the low—the strong—the light;
And, oh! it was a glorious sight
That strife of town and gown to see.

8-: From Reginald Dalton (Edinburgh: William Blackwood; London: T. Cadell – 1823), by the Scottish novelist, critic and biographer John Gibson Lockhart (1794-1854)—Context: Reginald Dalton, who is to be entered in the University, has just arrived at an inn, in Oxford:

Reginald, seeing that there was still fine moonlight, went to the window to peep out for a moment […]. He […] was leaning over the balcony, contemplating a noble Gothic archway on the other side of the lane, when several persons turned the corner from the street, some retreating, apparently, and others following; for, though none of them were moving at speed, there was opposition and anger in the tones of the voices.
“Say the word, then, speak it out,” cried one voice. “Say Town, d—— ye, or I’ll floor your carcase.”
“Gown or Town?” roared another; “speak, or by jingo——”
“Stand back, stand back, I say; halt, you knaves,” shouted a third—“I am a clergyman.”
Reginald thought it was certainly very like Mr Keith’s voice; but they were all on the dark side of the lane, and he listened for another moment.
“I am a clergyman, I am a priest, sirs,” was reiterated.
“A clergyman! Then the devil’s in’t if you’re not a gownsman—down with him, down with him, I say.”