This is definitely a tricky question, but in the end it comes down to the actions our users take, just like if they were using a tape recorder or Google Glass-like-device.
We'll make sure to notify our users of the relevant privacy laws in their current area, and we'll always encourage people to tell those around them when they have Retrospect turned on.
I like the idea, but there absolutely has to be some sort of indicator that you're interested in recording this convo (and the less invasive the better). How about linking it to calendar events, and/or locations? Facial rec could be an indicator for recording when you hit Glass integration.
We'll make sure to notify our users of the relevant privacy laws in their current area, and we'll always encourage people to tell those around them when they have Retrospect turned on.