Surprisingly, only a handful of fitting mathematical frameworks have been discovered. The first one is based on the intractability of factoring large numbers, and on the intractability of modular logarithms -- giving rise to RSA, and Diffie-Hellman, and ElGamal. Next to it stands the elliptic curves framework, based on the intractability of reversing "point addition" in the ECC way. And the third is lattice-based intractability hinged on the difficulty to find a good basis to express a multi-dimensional point, when the number of dimensions is high enough.
For the first framework the basic idea is to use substitution tables over a large alphabet. This is featured through modular arithmetics where all integers are mapped into a final set, large as it may be. By raising a number in the set to the power of selected number (let's call it the encryption key), one generates a substituted letter for it (from the same large alphabet), but the task of finding a power value to raise by the new letter and find the former, is highly intractable for our adversary. Namely having the power value to substitute a letter for another, does not lead to the power to reverse that scheme.