Growing on a Wall

I had a 1st this morning. My next door neighbours on the lane have primulas in flower. I walked along to take some macro shots. 
A fierce, officious woman quickly appeared asking : "can I help you ? '
" thank you no, I don't need help " I replied.
" what are you doing  ? ", she then asked.
"taking photo's of flowers in the sun "
" This is private property and we have vulnerable people here . Photography is not allowed "
"I'm sorry I'll go ".
Before you imagine me clambering over someone's garden, this is a public road way. No signs about anything.