Feeling a bit hopeless...

about ever learning to read music and play my mandolin.  How on earth do you remember which note is which and on which string, and where it is on the fret board?  And even when I manage to follow the notes, if I just look down for a second to check my finger position, I completely lose my place!  I can't imagine how people ever learn it...
I am enjoying trying though. I just hope the neighbours can't hear me... If they can, oh the shame! :-)