Origins
Every extreme is an edge. The word descends from Latin extrēmus, the superlative form of exterus ('outer'), built on ex ('out of'). To call something extreme is to place it at the outermost boundary of a scale.
Latin used extrēmus in both literal and figurative senses. The extrēmus fīnis was the farthest border of a territory. An extrēmum remedium was a last resort — the outermost option when all others had failed. The word carried a sense of finality: beyond the extreme, there is nothing.
Old French inherited it as extreme, and English adopted it in the 15th century. Early uses were spatial — the extreme north, the extreme edge of a cliff. The shift to intensity came quickly: extreme cold, extreme measures, extreme poverty.
Latin Roots
The family it belongs to is large. Exterior, external, extra, and extraneous all share the same Latin root. Even the prefix extra- (as in extraordinary) comes from the same source — literally 'beyond the outer'.
In mathematics, extrema are the maximum and minimum values of a function — the outermost points on a curve. The word has never lost its original meaning of standing at the very edge.