That’s exactly why op said we should abandon Unix time. The “who cares about leap seconds” standards is called TAI, and the difference between TAI and Unix time changes every time a leap second is introduced.
Unix time decrees every day has exactly 86400 seconds. But the introduction of leap seconds means that While most days have 86400, some earth days have 86401 seconds and potentially there could also be days with 86399
TAI makes sense everywhere, Unix time only on earth.
Number of seconds since a given time seems fine until we're travelling significantly outside of a common reference plane.