How would a rational autonomous system behave? What might go wrong and what can we do about it? Steve Omohundro discusses these questions in a thought-provoking talk:

(youtube)

Hat tip to Manu मनु for pointing me to this video in a comment.