The Squircle
It’s called a squircle, a cross between a circle and square. Unlike more typically rounded rectangles where the corner exhibits a perfect semi-circle, squircles have a more squished shape. The rounding does not suddenly begin, instead the curve starts deep within the straight line and gradually grows into a visible curve.
While rounded rectangles have a constant corner radius, the corner radius of a squircle decreases or tightens as the curve deepens, and then symmetrically expands again on the other side of the curve.
The mathematics behind this shape are well worth a read
Make it work in Figma
How to round box corners inside other rounded box?
Try visualizing a central point, the "origin", around which your first radius curves. Now use that same point to round your outer corner.
Form your curves this way and you’ll find the end result much easier on the eye.
Like all matters in design, there are no concrete rules - you’ll always find exceptions. It’s a question of instinct. It's up to you to decide what's best.
Sources
👉 Optical effects in user interfaces
👉 Rounded Corners in the Apple Ecosystem