IPv6 Only Domains

I recently added a new domain and added an IPv6 address. I then setup my Google Domain resource record AAAA to point to this. But the website does not load. Only after adding an IPv4 A record does the website load.

Do we have to use both in order for site to work, I would think that IPv6 only would be enough.