The phrase in a nutshell means in a few words; in brief or concise form.

This phrase originated in an allusion to a copy of the Iliad (a Greek epic poem, ascribed to Homer, telling how Achilles killed Hector at the climax of the Trojan War) which was supposedly small enough to be enclosed in the shell of a nut. The Roman statesman and scholar Pliny the Elder (23-79) told the story in his encyclopaedia of the natural and human worlds, Naturalis Historia (The Natural History – 77):

(translation: John Bostock & H.T. Riley – London, 1855)
Instances of acuteness of sight are to be found stated, which, indeed, exceed all belief. Cicero informs us that the Iliad of Homer was written on a piece of parchment so small as to be enclosed in a nut-shell. He makes mention also of a man who could distinguish objects at a distance of one hundred and thirty-five miles. M. Varro says that the name of this man was Strabo; and that, during the Punic war, from Lilybæum, the promontory of Sicily, he was in the habit of seeing the fleet come out of the harbour of Carthage, and could even count the number of the vessels. Callicrates used to carve ants and other small animals in ivory so minute in size that other persons were unable to distinguish their individual parts. Myrmecides also was famous in the same line; this man made, of similar material, a chariot drawn by four horses, which a fly could cover with its wings; as well as a ship which might be covered by the wings of a tiny bee.

In his translation (around 1387) of Polychronicon Ranulphi Higden maonachi Cestrensis, originally written in Latin by the Benedictine monk Ranulf Higden (circa 1280-1364), the Cornish author and translator John Trevisa (circa 1342-circa 1402) explained that Cicero

wroot alle þe gestes of Troye sotelliche, as it myȝte be closed in a note schale.
     in contemporary English:
wrote all the gests of Troye subtly, as it might be enclosed in a nutshell.

In the epistle dedicatory to his anti-theatrical pamphlet The Schoole of Abuse (1579), the English clergyman Stephen Gosson (1554-1625) also mentioned the story to suggest great condensation, brevity:

The title of my book doth promise much, the volume you see is very little […]. The Schoole which I builde, is narrowe, and at the first blushe appeareth but a doggehole; yet small Cloudes carie water; slender threedes sowe sure stiches; little heares haue their shadowes; blunt stones whette kniues; from hard rockes, flow soft springes; the whole worlde is drawen in a mappe; Homers Iliades in a nutte shell; a Kings picture in a pennie; Little Chestes may holde greate Treasure; a fewe Cyphers contayne the substance of a rich Merchant; The shortest Pamphlette may shrowde matter.

The English poet and playwright William Shakespeare (1564-1616) is the first known user of the image of the nutshell, without reference to the ancient story, as a type of something extremely small in The Tragedie of Hamlet, Prince of Denmarke (between 1599 and 1602):

(Folio 1, 1623)
– Hamlet. Denmark’s a Prison.
– Rosencrantz. Then is the World one.
– Hamlet. A goodly one, in which there are many Con-
fines, Wards, and Dungeons; Denmarke being one o’th’
worst.
– Rosencrantz. We thinke not so my Lord.
– Hamlet. Why then ’tis none to you; for there is nothing
either good or bad, but thinking makes it so: to me it is
a prison.
– Rosencrantz. Why then your Ambition makes it one: ’tis
too narrow for your minde.
– Hamlet. O God, I could be bounded in a nutshell, and
count my selfe a King of infinite space; were it not that
I haue bad dreames.

The earliest instance of the current phrase that I have found is from The Morning Post (London) of Monday 21^st February 1803; the review of The Hero of the North, a play which had premiered at Drury Lane the previous Saturday, contains:

The historical part of the plot every one knows. It lays in a nutshell.

 

Incidentally, an unattested Old-English form of nutshell, hnutsciell, apparently underlies Nursling, the name of a village in Hampshire first recorded in the late 8^th century as Nhutscelle (the suffix -ling was added later). According to Eilert Ekwall in The Concise Oxford Dictionary of English Place-Names (1960), the name was perhaps originally a jocular appellation given to a tiny abode or settlement.