Yes the difficulty adjustment references block timestamps, which ultimately come from external human time. But this does not alter the ontology of Bitcoin’s internal time. Timestamps are used to regulate rate, ensuring that block production tracks human chronological expectations. They do not define Bitcoin’s temporal substrate.
The smallest unit of temporal change in Bitcoin is still the block, an irreversible thermodynamic event created through proof-of-work. Whether those events occur faster or slower relative to wall-clock time is irrelevant: the ledger’s internal time advances only when a block is found. Timestamps adjust the equilibrium; blocks constitute time itself.