A 2048x2048 texture has roughly 4 Mio. pixels. Assuming you are using regular ARGB textures, with 8 bits per channel, that's 4 bytes per pixel. So in total that's 16 MB for the texture. Usually you also have mipmap levels, which adds another 33%, making the texture just over 21 MB in size.
According to the Steam hardware survey the average PC on Steam has about 2GB of video memory nowadays. That's enough for more than 90 such textures. A bit less in reality, as you can't use every last bit of VRAM for your textures. A high-end PC will have 8+ GB.
Using texture compression you could reduce texture size by a factor of 4, or even more, depending on the format.