Used for any large file on Freenet.

CHK keys are limited to 32KiB, so when larger files are inserted into Freenet, they must be split up and the individual pieces inserted as separate CHK keys. A metadata document called a splitfile is created which lists all the CHKs for individual pieces of the file, and that splitfile then becomes the actual key used to retrieve the file from Freenet.

Because individual blocks are unreliable, splitfiles use forward error correction (FEC).

During an insert, the data is first broken into 32KiB blocks, and then an equal number of check blocks are created to go with them.

To recover the original data, a combination of data and check blocks is required.

The node will request the data blocks and check blocks in random order until it has enough to reconstruct the original data.

Then it will reconstruct the original data, and return it to the requestor. At the same time it will re-encode (if necessary) and re-insert any blocks that it could not fetch (this is called splitfile healing).

The current implementation uses Reed-Solomon codes, implemented as Vandermonde codes by the Onion Networks FEC library. The RS codes can decode the k data blocks from any k of the n data blocks plus check blocks.

However, they are limited to 256 total data plus check blocks for the 8-bit word size used by Freenet. Moving to 16-bit words and longer codes would have a substantial performance penalty.

Therefore, large splitfiles are broken into multiple segments, each having a maximum of 128 data and 128 check blocks.

As of version 1255 of fred, cross-segment FEC is used, which aims to remove the problem of big files stalling on 99.9% waiting for last 1 or few segments. This important change should dramatically improve lifetime of keys, fixing a problem that was affecting mostly big files (over around 80MiB).