The phrasal verb shell out means to pay a specified amount of money, especially one regarded as excessive.

It is first recorded in Moral tales for young people (1801), by the Anglo-Irish novelist and educationist Maria Edgeworth (1768-1849):

“One of you, it’s plain, must shell out your corianders.”

(The word coriander (or coliander), short for coriander-seed (or coliander-seed), was slang for coin, money. The form coliander-seed, defined as meaning money, is first attested, in A New Dictionary of the Terms Ancient and Modern of the Canting Crew (1699), by “B. E. Gent.”.)

In Life in London; or, The Day and Night Scenes of Jerry Hawthorn, Esq. and his elegant Friend Corinthian Tom, accompanied by Bob Logic, the Oxonian, in their Rambles and Sprees through the Metropolis (1821), the sporting journalist and author Pierce Egan (1772-1849) quoted a “ballad, chaunted […] in all the richness of slang that characterizes the neighbourhoods of Nightingale-Lane and Tothill-Fields”:

O slumber, my Kid-wy, I no longer can stop,
For to-morrow poor Jemmy will be topp’d¹ on the drop²;
Though I’m napping my bib³, yet I hope he’ll die proud,
And all the prigs⁴ shell out⁵ to buy him a shroud!
¹ Hung
² Gallows
³ Crying, and wiping the eyes with an apron
⁴ Thieves
⁵ Subscribe, or club their pence together

In the same book, shell out is also used to mean to pay up:

‘Come, Covey,’ says I, ‘let’s have a kevarten [= a quarter of a pint] of gin;’ and, so my Lord, so help my bob it’s true, I gave the whole of the kevarten to Neddy. ‘Vy,’ says Dirty Suke, ‘it’s a mere thimblefull in his gills;’ and so it was, indeed, my Lord. ‘Give him another kevarten,’ says she, ‘and if you are too scaly to tip for it, I’ll shell out and shame you.’

FIRST EXPLANATION

It is often said that shell out has its origin in the literal sense, attested in the mid-16^th century, of shell (out), which is to remove (a seed) from its shell, husk, or pod. Figuratively therefore, to shell out money would be to remove its casing (purse, wallet, etc.) and hand over the contents.

However, this might be a folk etymology.

SECOND EXPLANATION

According to John Stephen Farmer (1854-1916) and William Ernest Henley (1849-1903) in Slang and its Analogues Past and Present (1903), the phrasal verb is from the plural noun shells, which was thieves’ cant for money. This use of shells is first recorded in A notable discouery of coosenage (1592), by the English writer Robert Greene (1558-92):

In Figging law
The Cutpurse, a Nix
He that is halfe with him, the Snap
[…]
The purse, the Bong
The monie, the Shels.

(The term figging law was defined as “the art of picking pockets” in A Classical Dictionary of the Vulgar Tongue (1785), by the English antiquary and lexicographer Francis Grose (1731-91).)

Robert Greene told the following story in The second part of conny-catching (1592):

A kinde conceipt of a Foist performed in Paules.
Ther walked in the middle walk a plain countrey farmar a man of good wealth, & that had a well lined purse, onely barely thrust vppe in a round slop which a crue of Foists hauing perceiued, ther harts were set on fire to haue it, and euery one had a fling at him but all in vaine, for he kept his hand close in his pocket, and his purse fast in his fist like a subtil churle, that either had béen forewarnd of Paules, or else had afore time smoakte some of that faculty, well how so euer, it was vnpossible to doe any good with him he was so wary. The Foists spying this, strained their wits to the highest string how to cōpasse this boung, yet could not all their polliticke conceipts fetch the farmar ouer, for iustle him, that with him, offer to shake him by the hand, all would not serue to get his hand out of his pocket. At last one of the crue that for his skil might haue bin Doctorat in his mistery, amongst them all chose out a good Foist, one of a nimble hand & great agility, and said to the rest thus: Masters it shal not be said such a base peasaunt shall slip away from such a crue of Gentlemen Foistes as wee are, and not haue his purse drawen, and therfore this time Ile play the staule my selfe, and if I hitte him not home, count me for a bungler for euer, and so he left them and went to the farmar and walkt directly before him & next him three or foure turnes, at last standing still he cryed alas honest man helpe me, I am not well, and with that suncke downe suddenly in a sowne, the poor Farmer seeing a proper yong gentlemā (as he thought) fall dead afore him, stept to him, helde him in his armes, rub’d him and chafte him: at this there gathered a greate multitude of people about him, and the whilst the Foiste drewe the Farmers pursse and awaye: by that the other thought the feate was done, he began to come somthing to himselfe again, and so halfe staggering, stumbled out of Paules, and went after the crue where they had apointed to méete, and there boasted of his wit and experience. The Farmer little suspecting this villanye, thrust his hand into his pocket and mist his pursse, searcht for it, but lyning and shelles and all was gon, which made the Country man in a great maze.