Methods for Compressing Saved Replays to Smaller Sizes?

Hi again. I have successfully created a replay system for my game, that works perfectly. However I have one problem. The files are way too big. I studied the size per second, about every 5 seconds is 150kb. Which means that 1 minute of recording is nearly 2mb in size, for 30mins its 50mb which is a big problem since I’m making an stg engine, and those tend to run about 30mins. I have studied other games such as Touhou, in which the replays are 50kb to 75kb in size.

Is there any way I could compress my replays? Specifically a method that doesn’t mess with the default system encryption format as I run my replays through that before saving.

What do you mean by “default system encryption”?

There are several things you could look at doing:

  • Store less data. Are you storing an int when the values never go above 255 (byte) or 65535 (ushort)? Are you storing a Matrix when you could really just be storing the rotation and position?

  • Are you storing data you don’t need? If firing a rocket, you could store just the launch event (position, direction), and the end result (player 2 was hit). The path the rocket takes between launch and the hit could be played as normal by the game, ignoring any collision results because the replay already says when something was hit by it.

  • Run length compression. If an entity is standing still, you do not need to store its position every frame if that position never changes.

  • Can you afford to lose precision without affecting playback too much? Can you store a 16-bit float instead of a full 32-bit float, with the resulting lower precision?

You could also try reducing your recording interval and use lerp (or a more advanced version) to calculate the bits in-between.

In on of my games I only record every 200ms and calculate all the other bits of data I need

I am currently storing player inputs from an enumerator each frame, using delimiters to separate frames, and then having C# parse the strings to figure out what enumerator is for what, and force these inputs to my virtual key manager class.

http://pastebin.com/gE94fFDy

here is a snippet of the replay data uncompressed from the System.Security.Cryptography namespace. I am guessing that, not saving the inputs that are FREE (which means the key isn’t being pressed), should reduce size.

But by how much is the question, though I really want to look into compression without changing the way my system works, this should be a reasonable change, especially since the player won’t logically do anything every frame.

You only need to store the data of keys that are pressed. If they are not pressed don’t save them.

Do this by having a list of keys. If no keys are pressed in a frame the list would be empty. If only 2 keys pressed only 2 records stored. Also have u made your enum a byte or left it a standard int?

The enums all return bytes from 0 - 12 for keys, and 0 - 3 for the key states. Key states are decided by how long the player has held a key.

My problem now, seeing that strings may not be suitable for small sizes, I want to just store data in place of the strings.

Is it possible to still use the delimiters as shown in my example? or if I were to use strings, should I just append the key number and state number from the enumeration? would that be enough?

My target file size for 30mins is atleast 2mb or less. As I said before, the current system could output files up to 50mb by the end of a full game run.

Why not saved The enum instead of strings?

I’m just not sure how to do it anymore. I’ve been trying all day but no luck, I was able to reduce the file size and predictions match my limit for the most part (about a 1mb over) but idk.

I have never done replays, and whenever I plop replays from other games into a text editor I see a garbage mess, obviously I can’t get any information out of that, so I don’t know where to start.

All I need is a way to save each frame, each key pressed, and each key state, and read that data back. But if its not some kind of string I’m helpless on how to do it.

Look at datacontract it allows u to save types to file. U can save arrays, classes etcs directly to file then reload them.

Thank you, this sounds like a life saver, I will definitely look into it and report back later.

If file size is your concern I wouldn’t recoment datacontractserializer, it writes a very verbose XML file.

Use the BinaryWriter to write the data in binary format.


example:
writer.Write(frame);
write.Write(shotState);
write.Write(bombState);

There are ways to compress that more. If you have only 4 states (2 bits) per key you can store 4 keys in one byte (8 bits) by shifting and ANDing the values. For 12 keys, that is 3 bytes.
https://www.google.com/search?q=c%23+bitwise+operations+on+enums
example:
writer.Write(frame);
write.Write((byte)(shotState >> 0 | bombState >> 2 | leftState >> 4 | rightState >> 6));

For an (int32) frame number + 3 bytes for states you have 4+3=7 bytes per frame, 760 bytes per second, 760*60 bytes per minute.
25kb per minute, not bad , huh?

You could skip writing the frame number on every recond, perhaps by writing frame # only every 600 frames (10 seconds).

Ah I see. I have never worked binary format ever, and the setup you mentioned seems abit confusing. Is there a general tutorial on how to build a replay system? I have searched but I can’t seem to find anything for XNA/Monogame specifically.

You don’t need to use a binary formatter. I don’t have time to write you some sample code but what I can do is post some of my current code.

I apologies for the lack of annotation I had to De-compile the binary file after data corruption and have only partial re-anotated it.
Now in the code below I have a class that I am saving and loading.
Now the class BuildingData is what i save to file
I specify the data to save with [datacontract] and [datamember] that is the only data i save

the other area to look at are SaveToFile() and LoadFromFile() the rest is just processing the data before saving and after loading.

Hope this helps you out a bit.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using BlockEmpire.Engine.World;

using System.IO;
using System.Runtime.Serialization;

using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;

namespace BlockEmpire.Engine.Design
{
public enum BuildningType { Housing, Rescource, UnitTraining, Defence };

public struct BuildingBlock
{
    public Point3D Location; //the locaiton of  the block
    public BasicBlock Block; //the block data
    ///wherether this block can be replaced with other blocks of same type
    public bool CanUseOtherBlocks;

    public BuildingBlock(Point3D location, BasicBlock block, bool canUseOtherBlocks)
    {
        Location = location;
        Block = block;
        CanUseOtherBlocks = canUseOtherBlocks;
    }
}

[DataContract]
public struct BuildingData
{
    [DataMember]
    public List<BuildingBlock> Data;
    [DataMember]
    public string Name; //optional variable for user
    [DataMember]
    public BuildningType Type;

    private bool _haveGeneratedGraphicAssets;//if we have generated the verts list and the HUD_Texture
    public bool HaveGeneratedGraphicAssets { get { return _haveGeneratedGraphicAssets; } }
    public List<VertexPositionColorTexture> Verts; //list of all verts for the building
    public VertexPositionColorTexture[] RenderVerts; //list of all verts for the building
    public Texture2D HUD_Texture; //the texture to draw in the menus
    private Point3D LastLoc;

    public BuildingData(BuildningType type)
    {
        Type = type;
        Name = "";
        Data = new List<BuildingBlock>();
        _haveGeneratedGraphicAssets = false;
        Verts = new List<VertexPositionColorTexture>();
        HUD_Texture = null;
        RenderVerts = null;
        LastLoc = Point3D.Error;
    }

    public void BlockChanged(Point3D blockLocation, BasicBlock block)
    {
        BuildingBlock b;

        if (block.Id == UserFriendlyBlockIds.Air)
        {
            //if (blockLocation.Y > 10)
                for (int i = 0; i < Data.Count; i++)
                {
                    b = Data[i];
                    if (b.Location.X == blockLocation.X &&
                        b.Location.Y == blockLocation.Y &&
                        b.Location.Z == blockLocation.Z)
                    {
                        Data.RemoveAt(i);
                        return;
                    }
                }

            return;
        }

        for (int i = 0; i < Data.Count; i++)
        {
            b = Data[i];
            if (b.Location.X == blockLocation.X &&
                b.Location.Y== blockLocation.Y &&
                b.Location.Z == blockLocation.Z)
            {
                b.Block = block;
                Data[i] = b;
                return;
            }
        }
        Data.Add(new BuildingBlock(blockLocation, block, true));
    }

    public void GenerateGraphicAssets()
    {
        Verts = new List<VertexPositionColorTexture>();

        List<BuildingBlock> renderBlocks = new List<BuildingBlock>();

        int lowest = int.MaxValue;
        int leftmost = int.MaxValue;
        int rightmost = int.MinValue;
        int nearset = int.MinValue;
        int farest = int.MaxValue;
        int centerx, centery;

        for (int i = 0; i < Data.Count; i++)
        {
            if (lowest > Data[i].Location.Y)
                lowest = Data[i].Location.Y;

            if (leftmost > Data[i].Location.X)
                leftmost = Data[i].Location.X;

            if (rightmost < Data[i].Location.X)
                rightmost = Data[i].Location.X;

            if (nearset < Data[i].Location.Z)
                nearset = Data[i].Location.Z;

            if (farest > Data[i].Location.Z)
                farest = Data[i].Location.Z;
        }

        centerx = (rightmost - leftmost) / 2 + leftmost;
        centery = (nearset - farest) / 2 + farest;
        for (int i = 0; i < Data.Count; i++)
        {
            BuildingBlock block = Data[i];
            block.Location.Y -= lowest;
            block.Location.X -= centerx;
            block.Location.Z -= centery;
            renderBlocks.Add(block);
        }


        for (int i=0; i < Data.Count; i++) //loop through all blocks
        {

            for (int side =0; side < 6; side++)
                BlockInfo.Blocks[(int)renderBlocks[i].Block.Id].GetSideVerts(side,
                new Vector3(renderBlocks[i].Location.X, renderBlocks[i].Location.Y, renderBlocks[i].Location.Z),
                ref Verts);

        }

        _haveGeneratedGraphicAssets = true;
    }

    public void RenderVertsAssets(ref Camera gameCamera, Point3D loc)
    {
        if (LastLoc.X != loc.X || LastLoc.Z != loc.Z || LastLoc.Y != loc.Y) //if it has moved
        {
            RenderVerts = new VertexPositionColorTexture[Verts.Count];
            for (int i = 0; i < Verts.Count; i++)
            {
                RenderVerts[i] = Verts[i];
                RenderVerts[i].Position.X += loc.X;
                RenderVerts[i].Position.Y += loc.Y;
                RenderVerts[i].Position.Z += loc.Z;

            }

        }
        foreach (EffectPass pass in gameCamera._effect.CurrentTechnique.Passes)
        {
            Engine.graphicsDevice.DrawUserPrimitives(PrimitiveType.TriangleList, RenderVerts, 0, Verts.Count / 3);
        }
        LastLoc = loc;
    }

    public void OnCreat()
    {

    }


    public void OnDestroy()
    {

    }
}

public class BuildingInterface
{
    private static List<BuildingData> _buildings;
    public static List<BuildingData> Buildings { get { return _buildings; } }

    public static BuildingData CurrentBuilding;

    public static void Save()
    {
        if (CurrentBuilding.Name == null || CurrentBuilding.Name == "")
            Guide.BeginShowKeyboardInput(Microsoft.Xna.Framework.PlayerIndex.One, "Enter Building Name", "Please enter the name you wish to call this building", "",
                OnKeyBoardExit, null);
        else
            Guide.BeginShowKeyboardInput(Microsoft.Xna.Framework.PlayerIndex.One, "Enter Building Name", "Please enter the name you wish to call this building", CurrentBuilding.Name,
                OnKeyBoardExit, null);
    }

    private static void OnKeyBoardExit(IAsyncResult r)
    {
        string result = Guide.EndShowKeyboardInput(r);
        if (result == null || result.Length < 3)
        {
            while (Guide.IsVisible) { }
            Guide.BeginShowMessageBox("Error Saving", "Design was not saved please enter a building name that is longer than 3 letters",
                new List<string> { "Ok" }, 0, MessageBoxIcon.Error, null, null);
            return;
        }
        CurrentBuilding.Name = result;
        PostSave();
    }

    private static int SaveLocation = -1;
    private static void PostSave()
    {
        SaveLocation = -1;
        if (_buildings == null)
            _buildings = new List<BuildingData>();
        for (int i =0;i < _buildings.Count; i++)
            if (_buildings[i].Name == CurrentBuilding.Name)
            {
                SaveLocation = i;
                _buildings[i] = CurrentBuilding;
                break;
            }

        if (SaveLocation == -1) //if curretn building does not extis
            _buildings.Add(CurrentBuilding);

        SaveToFile();
    }

    private const string FilePath = ".\\buildings.dat"; //the path for the building data

    /// <summary>
    /// save the build data to file
    /// </summary>
    private static void SaveToFile()
    {
        using (FileStream f = File.Create(FilePath))//create the file on pc
        {
            DataContractSerializer writer = new DataContractSerializer(typeof(List<BuildingData>)); //create the serializer
            writer.WriteObject(f, _buildings); //write to file
        }
    }


    /// <summary>
    /// load the building data from file
    /// </summary>
    private static void LoadFromFile()
    {
        if (_buildings == null)
            
            _buildings = new List<BuildingData>();

        _buildings.Clear();

        using (FileStream f = File.Open(FilePath, FileMode.OpenOrCreate))//create the file on pc
        {  
            DataContractSerializer reader = new DataContractSerializer(typeof(List<BuildingData>));//create the serializer
            _buildings = (List<BuildingData>)reader.ReadObject(f); //load from file

        }

    }


    public static void Load(ref Sector[,] sectors)
    {
        LoadFromFile();
        CurrentBuilding = _buildings[0]; //set the current building
        BuildingBlock block;
        Point3D blockpos;
        Point2D sectorpos;
        List<Point2D> reGenerations = new List<Point2D>();

        for (int i = 0;i < CurrentBuilding.Data.Count; i++)
        {
            block = CurrentBuilding.Data[i];

            sectorpos = Sector.CalculateSectorFromBlockCord(block.Location); //get the current sector
            blockpos = block.Location; //get the block
            blockpos.X -= sectorpos.X * Sector.Sector_Size; //workout the block location in sector
            blockpos.Z -= sectorpos.Z * Sector.Sector_Size; //workout the block location in sector

            sectors[sectorpos.X, sectorpos.Z].WriteblockIds(blockpos, block.Block.Id, block.Block.Data); //write the new block id
            if (!reGenerations.Contains(sectorpos)) //if this sectors has not been set to be regenerated
                reGenerations.Add(sectorpos); //add to list
        }

        for (int i=0;i< reGenerations.Count; i++) //regenerate all needed sectors
            sectors[reGenerations[i].X, reGenerations[i].Z].CalculateGeomotry();

    }
}

}

That method works but the file size is insanely high. If I were recording for an entire game run the size would be about half the size of the game itself. Is there a way to compress this down? So that the average size is at the most, 1mb in size for 30min recording. That’s my aim size.

EDIT: I ended up using the binary method, I was able to reduce the size by alot, with my current method of saving every frame, I can get about 1mb per 30mins saving binary files. If I only save frames where the player input keys, I can reduce it alittle more. Other than that, I am fine with what I have now.

I probably will remove the frame counter for the sake of keeping sizes down until I find a better way of saving. Thanks to everyone who contributed to helping me.