The dog’s letter is a name for the letter R, from its resemblance in sound to the snarl of a dog. It was so named after Latin canina litera, the canine letter, used by the Roman poet Persius (Aulus Persius Flaccus – 34-62); in his first Satire, he places in the mouth of his interlocutor the following arguments against the satirist: he hurts people’s feelings, important people close their doors to him, and he bites and snarls at his victims like a dog:

(from: Persius 1. 107-10, by William S. Anderson – The Classical Quarterly, November 1958)
sed quid opus teneras mordaci radere vero
auriculas? vide sis ne maiorum tibi forte
limina frigescant; sonat hic de nare canina
littera
     translation:
But what need is there to rasp upon tender little ears with biting truth? Be careful that the thresholds of the great do not perhaps grow cold towards you; here there is the nasal sound of the canine letter.

In French, la lettre canine seems to have first appeared, with reference to Persius (Perse in the text), in Champ Fleury. Auquel est contenu Lart & Science de la deue & vraye Proportion des Lettres Attiques, quon dit autrement Lettres Antiques, & vulgairement Lettres Romaines Proportionnees selon le Corps & Visage humain (1526), a treatise on typography by the French humanist, printer and librarian Geoffroy Tory (circa 1480-1533):

The dog’s letter was first used by the British poet and clergyman Alexander Barclay (circa 1484-1552) in The Shyp of Folys of the Worlde (1509), an adaptation of Daß Narrenschyff ad Narragoniam (1494), by the German humanist Sebastian Brant (1458-1521):

(1874 edition)
This man malycious whiche troubled is with wrath
Nought els soundeth but the hoorse letter R
Thoughe all be well, yet he none answere hath
Saue the dogges letter.
     in contemporary English:
This malicious man who is troubled with wrath
Sounds nothing else but the hoarse letter R
Though all be well, yet he has no answer
Save the dog’s letter.

In The English Grammar (1637?), the English poet and playwright Benjamin ‘Ben’ Jonson (circa 1573-1637) wrote:

(1640 edition)
                                                                                                                    R
Is the Dogs Letter, and hurreth in the sound; the tongue striking the inner palate, with a trembling about the teeth. It is sounded firme in the beginning of the words, and more liquid in the middle, and ends: as in
                                                                                              rarer, riper.
And so in the Latine.

In The Westminster Review (London) of April 1830, a critic lambasted Satan, a collection of poems by one R. Montgomery:

Mr. Montgomery begins well—Satan, by Montgomery, a bold and original authorship. Then turning the leaf, we are somewhat startled by these three words on the next page, To my Friend. As there is only the difference of the dog’s letter between friend and the quality of the subject, we looked to the Errata, thinking it probable there was a misprint of fiend; but as none is acknowledged, we suppose the friend is one whom it is not decorous more distinctly to particularize. It is the fashion of the day to make biography a work of friendship. Moore writes the life of Byron; Campbell is the historian of Lawrence; Paris takes the life of Davy, and Mr. Montgomery handles Satan. Indeed, on looking again at the Address, we discover the ingenuity of the device, on one page stands “To my Friend,” “Satan, Book 1st,” is the next title, completing the Dedication. As thus, To my Friend Satan his First Book. Horace instructs us that neither gods nor man endure mediocre poetry, and consequently Mr. Montgomery had no course, but to address his song to the third estate, whose liberal patronage of every thing bad may reasonably be reckoned on. The arch enemy’s ear for discord must needs be gratified by such verses as we see before us.