Ghosts of old loves

[Xinfu]

"'It is simplicity itself,' said I. 'When you bared your arm to draw that fish into the boat I saw that J. A. had been tattooed in the bend of the elbow. The letters were still legible, but it was perfectly clear from their blurred appearance, and from the staining of the skin round them, that efforts had been made to obliterate them. It was obvious, then, that those initials had once been very familiar to you, and that you had afterwards wished to forget them.'

"What an eye you have!" he cried, with a sigh of relief. 'It is just as you say. But we won't talk of it. Of all ghosts the ghosts of our old lovers (some versions have loves, not lovers) are the worst. Come into the billiard-room and have a quiet cigar.'
-------------------------------
What do the red words mean? According to context, do they mean that old memories are always scary, haunting us like ghosts?

[poli]

Metaphorical ghosts are no necessarily scary. It can mean regretful memories in this case, or longing for something no longer attainable.

[Xinfu]

Metaphorical ghosts are no necessarily scary. It can mean regretful memories in this case, or longing for something no longer attainable.

Do you mean 'old loves' are the things we once wanted to get?

[Rusty]

old loves = old routines/habits/addictions

[Xinfu]

um...if GHOST means memories, would it not mean of all memories, the memory of past things is the worst? If so, then do you think it's not logical? because memory must be related to the past; there could not be any other memory.

[Rusty]

Here, the word 'ghosts' is being associated with memories.

Of all memories, the memories of our past loves/lovers are the worst.

The thing that perhaps you're missing is that the memory, in this case, a past love, is a bad memory.
That is why the tattoo was mentioned. The letters in the tattoo belonged to a past love that the wearer of the tattoo wanted very much to forget.

[Xinfu]

Here, the word 'ghosts' is being associated with memories.

Of all memories, the memories of our past loves/lovers are the worst.

The thing that perhaps you're missing is that the memory, in this case, a past love, is a bad memory.
That is why the tattoo was mentioned. The letters in the tattoo belonged to a past love that the wearer of the tattoo wanted very much to forget.

By the red words, do you mean a relationship with a woman?

[Rusty]

Yes.

[Xinfu]

Thank you.