I'm surprised that you don't get 80+ FPS with your GPU, especially since you're playing on mid-high settings.
P.S.
Did the 5.5GB VRAM usage happen when you played a game in ultra, or were the settings on high?

Usually I have Vsync on as I only have a 60Hz monitor, and the settings where on high, the most recent triple A game I can think of was Shadow of the Tomb raider, recently I've been playing a lot of non-demanding games.

I haven't played AAA games in a long time, but unless something major has changed, since you're using Vsync your GPU won't be used to it's maximum potential so your max VRAM usage might be lower than what it could be without Vsync.

Also Assassin's Creed Valhalla

AC Valhalla is like how many games? I guess the number is one. And Ubisoft aging engine.. so no, not a factor.

By benchmarking it. Because once usage exceeds VRAM on card FPS drops like rock. Basically first you of course need to benchmark in non VRAM restricted way, then you get expected percent difference between two cards. Then once that difference suddenly gets huge deviation you know you are VRAM limited. And there are examples of that. The easiest one being RTX3080 being 2x faster than RTX2080.

When nVidia made claim of that, they tested it at 4k Ultra Nightmare setting which uses about 9GB of VRAM, exceeding 2080 8GB of VRAM so for example it did 88FPS average versus 3080 189FPS, which is 118% increase. But once you drop testures to ultra, which makes no performance difference it is 111FPS vs 189FPS, 70% increase. And textures outside VRAM have no performance impact.

This is based on Hardware Unboxed review of RTX3080 in case you want to see full numbers. But VRAM does matter, since if usage exceeds it, card needs to swap things with system RAM which is much slower. And will drop performance outside normal scaling. While exceeding it a little is not end of the world, it will only get worse as games become more demanding.

@Seth22087 You can´t "benchmark VRAM actual usage. What Hardware Unboxed described is literally bandwidth limitation. If it can´t provide so much data as the game needs the framerate is not that high, and if it is way less, the VRAM will overflow and stutter will be an issue apart from lower framerate.

How do I know this because I am using a card that is vram limited in many games. Second you are wrong about why it stutters.
Reason for stuttering is because game cannot load all the data that it needs into the VRAM. So instead of giving

an error, the extra data is kept in system RAM and streamed into the VRAM as required. Unfortunately this streaming process is a lot slower than accessing it directly from VRAM so the GPU often has to wait for the data to arrive in VRAM

before it can continue to process the graphics. This holdup shows up as stuttering and performance drops in games and often various texture bugs too. BTW I am a soft developer I literally write codes for living but that's irrelevent stuff.

Anyway, i´m tired of trying to give the real explanation for this since the Nvidia cards launched. We´ll see in 3-4 years if that capacity is indeed an issue or not. I already know the answer, but everyone can believe what they want.

No need to be sorry I wasn't offended by anything. I am understanding your point of fast bandwidth but for that to work whole architecture needs to be changed but for current situation we need more vram to compensate for problem.

That´s completely true. In the end, is the use you´re going to give it. I was just stating for gaming purposes bandwidth>VRam capacity, always, as you need to stream large amounts of data to achieve higher framerates. For rendering or graphic design usually Vram>Bandwith as you require large amounts of data stored.