The CryENGINE streaming engine takes care of the streaming of meshes, textures, music, sounds, animations, etc.
The file IStreamEngine.h in CryCommon contains all the important interfaces and structs used by the rest of the engine.
First of all there is the IStreamEngine itself. There is only one IStreamingEngine in the application and it controls all the possible I/O streams. Most of the following information comes directly from the documentation inside the code, so it's always good to read the actual code in IStreamEngine.h file for any missing information.
The most important function in IStreamEngine is the StartRead function and is used to start any streaming requests.
UNIQUE_IFACE struct IStreamEngine
{
public:
// Description:
// Starts asynchronous read from the specified file (the file may be on a
// virtual file system, in pak or zip file or wherever).
// Reads the file contents into the given buffer, up to the given size.
// Upon success, calls success callback. If the file is truncated or for other
// reason can not be read, calls error callback. The callback can be NULL
// (in this case, the client should poll the returned IReadStream object;
// the returned object must be locked for that)
// NOTE: the error/success/progress callbacks can also be called from INSIDE
// this function
// Return Value:
// IReadStream is reference-counted and will be automatically deleted if
// you don't refer to it; if you don't store it immediately in an auto-pointer,
// it may be deleted as soon as on the next line of code,
// because the read operation may complete immediately inside StartRead()
// and the object is self-disposed as soon as the callback is called.
virtual IReadStreamPtr StartRead (const EStreamTaskType tSource, const char* szFile, IStreamCallback* pCallback = NULL, StreamReadParams* pParams = NULL) = 0;
};Here are the currently supported streaming task types, and this enum should be extended if a new object type wants to be streamed:
enum EStreamTaskType
{
eStreamTaskTypeCount = 13,
eStreamTaskTypePak = 12, // Pak file itself
eStreamTaskTypeFlash = 11, // Flash file object
eStreamTaskTypeVideo = 10, // Video data (when streamed)
eStreamTaskTypeReadAhead = 9, // Read ahead data used for file reading prediction
eStreamTaskTypeShader = 8, // Shader combination data
eStreamTaskTypeSound = 7,
eStreamTaskTypeMusic = 6,
eStreamTaskTypeFSBCache = 5, // Complete FSB file
eStreamTaskTypeAnimation = 4, // All the possible animations types (dba, caf, ..)
eStreamTaskTypeTerrain = 3, // Partial terrain data
eStreamTaskTypeGeometry = 2, // Mesh or mesh lods
eStreamTaskTypeTexture = 1, // Texture mip maps (currently mip0 is not streamed)
};A callback object can be provided to the StartStream function to be informed when the streaming request has finished. The callback object should implement the following 2 function:
class IStreamCallback
{
public:
// Description:
// Signals that reading the requested data has completed (with or without error).
// This callback is always called, whether an error occurs or not, and is called
// from the async callback thread of the streaming engine, which happens
// directly after the reading operation
virtual void StreamAsyncOnComplete (IReadStream* pStream, unsigned nError) {}
// Description:
// Same as the StreamAsyncOnComplete, but this function is called from the main
// thread and is always called after the StreamAsyncOnComplete function.
virtual void StreamOnComplete (IReadStream* pStream, unsigned nError) = 0;
};Some extra optional parameters can also be provided when starting a read request:
struct StreamReadParams
{
public:
// The user data that'll be used to call the callback.
DWORD_PTR dwUserData;
// The priority of this read
EStreamTaskPriority ePriority;
// Description:
// The buffer into which to read the file or the file piece
// if this is NULL, the streaming engine will supply the buffer.
// Notes:
// DO NOT USE THIS BUFFER during read operation! DO NOT READ from it, it can lead to memory corruption!
void* pBuffer;
// Description:
// Offset in the file to read; if this is not 0, then the file read
// occurs beginning with the specified offset in bytes.
// The callback interface receives the size of already read data as nSize
// and generally behaves as if the piece of file would be a file of its own.
unsigned nOffset;
// Description:
// Number of bytes to read; if this is 0, then the whole file is read,
// if nSize == 0 && nOffset != 0, then the file from the offset to the end is read.
// If nSize != 0, then the file piece from nOffset is read, at most nSize bytes
// (if less, an error is reported). So, from nOffset byte to nOffset + nSize - 1 byte in the file.
unsigned nSize;
// Description:
// The combination of one or several flags from the stream engine general purpose flags.
// See also:
// IStreamEngine::EFlags
unsigned nFlags;
};The return value of the StartRead function is a IReadStream object which can be optionally stored on the client. The IReadStream object is refcounted internally. When the callback object can be destroyed before the reading operation is finished then the readstream should be stored somewhere, and abort needs to be called on it. This will take care of cleaning up the complete read request internally, and will also call the async and sync callback functions.
The Wait function can be used to perform a blocking reading requests on the streaming engine. This function can be used from an async reading thread, which wants to use the cryengine streaming system to perform the actual reading.
class IReadStream : public CMultiThreadRefCount
{
public:
virtual void Abort() = 0;
virtual void Wait( int nMaxWaitMillis=-1 ) = 0;
};The CryENGINE StreamEngine uses extra worker and IO-Threads internally. For every possible IO input a different StreamingIOThread is created which can run independent from each other.
Currently it has the following IO threads:
When a reading request is made on the streaming engine, it first of all checks which IO thread to use, and computes the sortkey. The request is then inserted into one of the IOThreads.
After the reading operation is finished the request is forwarded to one of the decompression threads (if the data was compressed), and then into one of the async callback threads. The amount of async callback threads is dependent on the platform, and some async callback threads are reserved for specific streaming requests types such as geometry and textures. After the async callback has been processed, the finished streaming request is added on to the streaming engine to be processed on the main thread. The next update on the streaming engine from the main thread will then call the sync callback (StreamOnComplete) and clean up the temporary allocated memory if needed.
For information regarding the IO/WorkerThreads please check the StreamingIOThread and StreamingWorkerThread class.
Requests to the streaming engine are not processed in a the same order as which they have been requested. The system tries to internally 'optimize' the order in which to read the data, to maximize the read bandwidth.
When reading data from an optical disc , it is very important to reduce the amount of seeks (and also from an HDD but to a lesser extent). A single seek can take over a 100msec, while the actual read time would be only a few msecs. Some official statistics from the 360 XDK:
The internal sorting algorithm takes the following rules into account in this order:
For Crysis 2 the following order was used: sounds/music, animations, meshes, low mip textures, high mip textures. (check image of actual disc layout below)
For information regarding the sorting, please refer to the source code in StreamAsyncFileRequest::ComputeSortKey(). Here is the essential sorting code:
void CAsyncIOFileRequest::ComputeSortKey(uint64 nCurrentKeyInProgress)
{
.. compute the disc offset (can be requested using CryPak)
// group items by priority, then by snapped request time, then sort by disk offset
m_nDiskOffset += m_nRequestedOffset;
m_nTimeGroup = (uint64)(gEnv->pTimer->GetCurrTime() / max(1, g_cvars.sys_streaming_requests_grouping_time_period));
uint64 nPrioriry = m_ePriority;
int64 nDiskOffsetKB = m_nDiskOffset >> 10; // KB
m_nSortKey = (nDiskOffsetKB) | (((uint64)m_nTimeGroup) << 30) | (nPrioriry << 60);
}The streaming engine can be polled for streaming statistics using the GetStreamingStatistics() function.
Most of the statistics are divided in 2 groups, one collected during the last second and another one from the last (which usually happens during level loading), but can also be force reset in game.
The Media Type info gives information per IO input system (HDD, Optical, Memory).
struct SMediaTypeInfo
{
// stats collected during the last second
float fActiveDuringLastSecond;
float fAverageActiveTime;
uint32 nBytesRead;
uint32 nRequestCount;
uint64 nSeekOffsetLastSecond;
uint32 nCurrentReadBandwidth;
uint32 nActualReadBandwidth; // only taking actual reading time into account
// stats collected since last reset
uint64 nTotalBytesRead;
uint32 nTotalRequestCount;
uint64 nAverageSeekOffset;
uint32 nSessionReadBandwidth;
uint32 nAverageActualReadBandwidth; // only taking actual read time into account
};The Request Type info gives information about each streaming request type (geometry, textures, animations, ...)
struct SRequestTypeInfo
{
int nOpenRequestCount;
int nPendingReadBytes;
// stats collected during the last second
uint32 nCurrentReadBandwidth;
// stats collected since last reset
uint32 nTotalStreamingRequestCount;
uint64 nTotalReadBytes; // compressed data
uint64 nTotalRequestDataSize; // uncompressed data
uint32 nTotalRequestCount;
uint32 nSessionReadBandwidth;
float fAverageCompletionTime; // Average time it takes to fully complete a request
float fAverageRequestCount; // Average amount of requests made per second
};The global statistics containing all the merged data:
struct SStatistics
{
SMediaTypeInfo hddInfo;
SMediaTypeInfo memoryInfo;
SMediaTypeInfo opticalInfo;
SRequestTypeInfo typeInfo[eStreamTaskTypeCount];
uint32 nTotalSessionReadBandwidth; // Average read bandwidth in total from reset - taking full time into account from reset
uint32 nTotalCurrentReadBandwidth; // Total bytes/sec over all types and systems.
int nPendingReadBytes; // How many bytes still need to be read
float fAverageCompletionTime; // Time in seconds on average takes to complete read request.
float fAverageRequestCount; // Average requests per second being done to streaming engine
int nOpenRequestCount; // Amount of open requests
uint64 nTotalBytesRead; // Read bytes total from reset.
uint32 nTotalRequestCount; // Number of request from reset to the streaming engine.
uint32 nDecompressBandwidth; // Bytes/second for last second
int nMaxTempMemory; // Maximum temporary memory used by the streaming system
};Different types of debug info can be requested using the following CVar: sys_streaming_debug x.
As mentioned earlier it is very important to minimize the seeks and seek distances when reading from an optical media drive, therefore the build system is slightly modified to optimize the internal data layout for streaming.
The easiest and fastest win is to not do any IO at all, but read the data from compressed data in memory. For this, small paks for startup and per level are created. These are loaded during level loading into memory and some stay there until the end of the level. Others are only used to speed up the level loading. All small files and small read requests should ideally be diverted to these paks.
A special RC_Job build file is used to generate these paks: Bin32/rc/RCJob_PerLevelCache.xml. These paks are generated during a normal build pipeline. The internal managment in the engine is done by the CResourceManager class, which uses the global SystemEvents to preload or unload the paks.
Currently the following pak's are loaded into memory during level loading (sys_PakLoadCache):
These paks are cached into memory during the level load process (sys_PakStreamCache):
Without these paks level loading can take up to a few minutes and streaming performance is reduced a lot, so be sure that they are available!
The information regarding all the resources of a level are stored in the resourcelist.txt and auto_resourcelist.txt. These files are generated by an automatic testing system which loads each level and executes a prerecorded playthrough on it. These resourcelist files are used during the build phase to generate the level paks.
All the data which is not in these in memory paks is performing actual IO on the optical drive or HDD, and it is also best to reduce the amount of seeks here. This optimization phase is also performed during the build process using the RC.
All the data which can be streamed is extracted from all the resourcelists from all the levels, and is removed from the default pak files (Objects.pak, Textures.pak, Animations.pak, ...) and put into new optimized paks for streaming inside a streaming folder.
The creating of the streaming paks uses the following rules:
The actual sorting code is hardcoded in the RC, and can be found at: Code\Tools\RC\ResourceCompiler\PakHelpers.cpp.
It is very important that only a single thread accesses the IO devices at the same time. If multiple threads are reading from the same IO device concurrently, then the reading speed is more than halved, possibly taking seconds to read just a few KB's!
The reason is that the IO reading head will partially read a few KB's for one thread, and then reads another few KB's for another thread, while always performing the expensive seeks in between.
The solution is to exclusively read from StreamingIOThreads during gameplay. CryENGINE will by default show an 'Invalid File Access' warning in the top left corner when reading data from the wrong thread and stall for 3 seconds to emulate the actual stall when reading from an optical drive.
For more information on how to track and fix 'Invalid File Accesses' in the game check the following confluence page: http://confluence/display/RND/Tracking+File+Access
A lot of Xbox 360 system have an internal HDD, but not all users take the time or know how to fully install the game to the HDD. That's why Microsoft has 2 GB of the internal HDD reserved for game cached data which is shared between all games. It is up to the developer to copy data into this cache to improve the loading or streaming times.
CryENGINE makes heavy use of this internal cache. It detects if there is an HDD available, and if the game has not been fully installed to the HDD yet. If so, then it will copy the streaming paks in the background to the HDD. The copying process is only activated when the optical drive is idle, to not decrease the normal streaming perforce. While copying the streaming paks to the cache, the highest mip streaming is disabled in the texture streaming system to increase the speed for the shadow copying process.
The average size of the streaming data for Crysis 2 was around 1.6GB and took a good 5 minutes to copy this data in the background to the HDD, which was very acceptable.
The implementation details of the Shadow Copying process can be found in the CPlatformOS_Xenon class.
For more detailed information regarding the 360 specifics please refer to the XDK white paper: 'Effective Use of Game Disc File Caching' or the XDK documentation.
The PS3 always has a hard drive available, so the game is almost fully installed to the HDD in the main menu. This makes the global streaming performance on the PS3 a lot better, but the actual HDD is rather slow. So on PS3 implementation makes heavy usage of streaming from both the HDD and Blu-Ray at the same time. All the video, audio and localization data is always streamed from the Blu-Ray for example.
The threading details of the streaming engine internally are also slightly different because all streaming system requests have to run on the PPU threads.
It is very easy to extend the current streaming functionality using the StreamEngine. A small example class is explained below on how to add a new streaming type.
Create a class which derives from the IStreamCallback (used to inform of streaming completion) and add some basic functionality to read a file. The file can either be read directly or using the streaming engine. When the data is read directly, it calls the ProcessData function to actually parse the loaded data. The function is also called from the async callback. Some processing can be performed here on the data if needed, because it's not running on the main thread.
The default parameters are used when starting a reading request on the streaming engine. It is also possible to provide the final data storage itself. This helps to reduce the amount of dynamic allocations performed by the streaming engine.
The class also stores the read stream object, to get information about the streaming request, or to be able to cancel it when the callback object is destroyed. The pointer is reset in the sync callback because after this call it won't be referenced anymore by the streaming engine.
#include
class CNewStreamingType : public IStreamCallback
{
public:
CNewStreamingType() : m_pReadStream(0), m_bIsLoaded(false) {}
~CNewStreamingType()
{
if (m_pReadStream)
m_pReadStream->Abort();
}
// Start reading some data
bool ReadFile(const char* acFilename, bool bUseStreamingEngine)
{
if (bUseStreamingEngine)
{
StreamReadParams params;
params.dwUserData = eLoadFullData;
params.ePriority = estpNormal;
params.nSize = 0; // read the full file
params.pBuffer = NULL; // don't provide any buffer, but copy data when streaming is done
m_pReadStream = g_pISystem->GetStreamEngine()->StartRead(eStreamTaskTypeNewType, acFilename, this, ¶ms);
}
else
{
// old way of reading file in a sync way (blocking call!)
const char* acData = 0;
size_t stSize = 0;
.. read file directly using CryPak or fopen/fread
ProcessData(acData, stSize);
m_bIsLoaded = true;
}
return m_bIsLoaded;
}
// Check if the data is ready and loaded
bool IsLoaded() const { return m_bIsLoaded; }
protected:
// implement the IStreamCallback function
void StreamAsyncOnComplete(IReadStream* pStream, unsigned nError)
{
if(nError)
{
return;
}
const char* acData = (char*)pStream->GetBuffer();
size_t stSize= pStream->GetBytesRead();
ProcessData(acData, stSize);
m_bIsLoaded = true;
}
void StreamOnComplete (IReadStream* pStream, unsigned nError)
{
m_pReadStream = 0;
}
// process the actual loaded data
void ProcessData(const char* acData, size_t stSize);
// store the stream callback object to be sure it can be canceled when the object is destroyed
IReadStreamPtr m_pReadStream;
// Extra flag used to check if the data is ready
bool m_bIsLoaded;
}