Origins
Leisure is not, at root, about relaxation. It is about permission. The word descends from Old French leisir, from Latin licēre — 'to be permitted, to be allowed'. To have leisure was to have permission to spend your time as you chose.
The semantic chain is elegant. Latin licēre meant 'to be lawful, to be permitted'. Old French leisir shifted the emphasis from legal permission to practical freedom: if you were permitted, you had time. If you had time, you were at ease. By the time the word reached English in the 14th century, the permission was assumed and the ease was paramount.
The Latin root left other descendants with very different personalities. Licence is permission to act — still close to the original. Licit means legally permitted. Illicit means forbidden, not permitted. Leisure and illicit are therefore opposites from the same root: one is the enjoyment of permission, the other its violation.
French Influence
The pronunciation splits along Atlantic lines. British English favours /ˈlɛʒ.ər/, rhyming roughly with 'measure'. American English prefers /ˈliː.ʒɚ/, rhyming with 'seizure'. Both are legitimate descendants of the Old French pronunciation.
The phrase 'at your leisure' preserves the oldest sense most accurately: do it when you are permitted, when circumstances allow — there is no rush, because the permission is open-ended.