The whole thing with the letters might be just about Leland.
Let's rewind a bit to the time when Leland was just a little boy at the Pearl Lakes. There was this man who Leland knew as mr. Robertson. He might've been just an average Joe just like Leland, but with "a thing" to little boys. Obviously just as Laura saw Leland as Bob, Leland saw that man as Bob too because he was either gifted or damned to see beneath the mask to the core of the evil that men do. Robertson himself might've been molested by someone else in his youth and that might've made him to be the man who eventually harmed Leland.
So, Leland had this incident in his past. It affected him so much that he developed a sexual problem which, at first, was shown as Leland using prostitutes. The the problem grew bigger. He had a thing for his daughter and so he started to molest her. The problem eventually made him to kill Teresa and later his own daughter (and eventually Maddy too).
Now, serial killers sometimes leave signs and clues behind when they are doing their dirty job. Sometimes it's about a cry for help. In this case, Leland might've tried to move the responsibility away from him. He started to spell the name of the man who "made" this problem to Leland. That's one theory.
Just yesterday I watched this movie called Session 9 and to large extent it was about personal disorder. One man listened to some tapes where a psychiatrist talked to a woman with multiple personalities. The evil personality in the woman was called Simon. The woman didn't know Simon existed but another personality knew him. In the end when the psychiatrist asked where does Simon live, the answer was something like this: "in the weak and wounded." (I can't remember the exact quote but I'm sure you get my point) It could be interpreted so that this kind of "evil" personalities in general appear in people who've been hurt very bad and who are weak. If the world of Session 9 was in the same world where Twin Peaks is, there would be people who would be able to see that personality as another person.
So in Twin Peaks Bob was kind of that Simon in Session 9 but in Twin Peaks the makers had created "an explanation" to the very source of this power that allows people to have that kind of mental problems. The question is, does the man with multiple personalities really control those personalities or do the personalities control the man. Whatever the answer is, the personality is connected to the personal history of the one who has that personality. In Leland's case, he had had this encounter with the man at the Pearl Lakes and this history showed as Leland (or Bob) "revealing" a part of it by putting the letters under the fingernails of his victims.
So, it might not be as simple as Bob's odd maneuver to put letters under the fingernails. It's not necessarily Bob's habit. It might be more about Leland. If the "Bob-personality" would be inside some other man, there might not be letters under the fingernails. But then again, there might be but it wouldn't necessarily try to spell ROBERT. Like, in Coop's case there could be EARLE or something like that (on the other hand, Coop could do ROBERT too as it is a part of his history)
But then again, I could be really really wrong :)

