How to load in an enormous amount of small images?

Content
,
1

Hello,

I am trying to write a program that can classify an image of a digit. I want to load about 60k images of 28x28 into memory so i can try training the underlying AI. How can i efficiently load these images? They are all grayscale.

I have tried just putting them all in the Content.mgcb file, but everytime i change something, it takes a very long time for the program to load. Is there perhaps some way of getting the .mgcb file to stop processing these images?

I have also tried just keeping the assets in the same directory as the program.cs file and using Texture2D.FromStream, but when i run it, the folder with the images does not get copied/moved to the directory that the .exe file ends up in and thus creates an error when trying to access any of the images.

So, does anyone know of a better way to do this?

2

Use an atlas image. An atlas is basically a huge image with a lot of images inside. You can create it for example with TexturePacker app. It will export a large PNG file and an xml file with the contained images information. You will end up just adding 1 PNG and 1 xml file into the MGCB.

3

In addition to this, because…

… you could probably compress further by overlaying four images into one using the RGBA channels. Doing this you could fit roughly 6840 images into a single 1920x1080 texture. You’d still need around 9 of these textures to hold your 60k images, but that might speed things up a bit. I’m not sure if it’s supported, but you could also look at using separate threads to stream this in, but I think the drive might be a bit of a bottleneck (if the Graphics object itself isn’t).

One way or the other though, you’re loading a massive amount of data into memory. That’s going to take some time.

For this, I can think of a couple of options…

  1. Add the image files to your project, just put them in some folder. Once in your project, look at the properies of the file. There should be a type or something like that. I think you have to set it to “Content” so that the build system will copy it to the output folder. Play around with this. I know it can be done I’m just not sure of the specifics.
  2. Put all the images in some folder, then add a post-build step that will copy all of the image files to the output folder. Just look into Visual Studio post-build and you should find the info you need. This is a project build setting that you can access via the project properties in visual studio and are just command prompt commands that are executed after a succcessful build. VS offers you a few aliases to get information, such as the output folder.
1 Like