rust-brotli

What's new in 9.0.0

• Custom dictionaries. Depends on brotli-decompressor 6.0.0, whose dictionary support is re-exported from this crate, so brotli::Decompressor, brotli::DecompressorWriter and brotli::BrotliState all gain attach_dictionary and attach_serialized_dictionary:
  • attach_dictionary takes a raw LZ77 prefix dictionary and is the equivalent of the C BrotliDecoderAttachDictionary with BROTLI_SHARED_DICTIONARY_RAW. Up to 15 may be attached, and they must be attached before the first read or write.
  • Dictionaries now live in their own buffers (the C implementation's "compound dictionary" scheme) rather than being copied into the ring buffer, so they stay addressable after the ring buffer wraps. Streams where output plus dictionary exceed the window decode correctly, and dictionaries larger than the window are no longer truncated.
  • attach_serialized_dictionary accepts the serialized shared dictionary container (magic 0x91 0x00) from draft-vandevenne-shared-brotli-format, including embedded LZ77 prefix dictionaries, custom word lists, custom transform lists and context-based dictionary selection.
• Breaking: the allocator crates move a major version, to alloc-no-stdlib 3.x and alloc-stdlib 0.3.x. alloc-no-stdlib's Allocator/SliceWrapper traits are part of the public API of this crate and of brotli-decompressor, so a crate that passes its own allocator to rust-brotli must move to the same generation: with a split, an allocator built against 2.x no longer satisfies the 3.x Allocator bound. The dependency ranges are pinned so that the resolver cannot span the major.
• Breaking, FFI: BrotliDecoderSetParameter now returns BROTLI_BOOL and takes the parameter selector as an int, matching the prototype in c/brotli/decode.h and the C library. It returns false for an unrecognized selector instead of forming an invalid enum discriminant at the ABI boundary. C callers that ignored the return value are unaffected; Rust callers of CBrotliDecoderSetParameter should pass BrotliDecoderParameter::... as i32.
• New non-default portable-float feature that makes encoder output target-independent, so the same input compresses to byte-identical output on every target. It routes the cost estimators' log2/powf onto the table-driven implementations no_std builds already use, instead of the platform libm, whose one-ULP differences could move a block-split boundary. It is off by default because it changes the emitted bytes and costs about 0.006% of ratio.
• The safe feature is now declared in Cargo.toml. The cfg had been referenced from src/ffi since the FFI was added but was never listed, so it could not actually be turned on. With it, the FFI drops the two unsafe impl Send lines and the encoder and multicompress entry points that rely on them; the decoder FFI is unaffected.
• Fix #257: SendableMemoryBlock<T> in the FFI allocator no longer claims Send for a T that is not itself Send.
• Fix #258: BroCatli recognizes a complete empty stream as soon as its window, ISLAST and ISLASTEMPTY bits are buffered, rather than waiting for the normal four- or five-byte prefix.
• The simd feature builds on current nightly again (Mask::to_int was renamed to Mask::to_simd).
• The seccomp feature builds again: the calloc-backed allocator macros are brought into scope in the brotli binary, and alloc-no-stdlib's unsafe feature, which gates them on 3.x, is enabled by it.

What's new in 8.0.4

Fix: adjust versions of rust-decompressor and rust-alloc-no-stdlib and alloc-stdlib so the Allocator<> trait is identical for all associated crates. Return BrotliFileNotCraftedForConcatenation when a new stream header advertises more whole source bytes than have been buffered. This prevents the unsigned subtraction in shift_and_check_new_stream_header from underflowing on truncated metadata headers. Return NULL from BrotliEncoderCreateInstance and BrotliEncoderCreateWorkPool when a caller-provided allocator returns NULL, rather than writing state through a NULL pointer. Wrap the mutable Broccoli FFI entry points in a local catch_unwind helper, matching the encoder FFI convention so Rust panics do not unwind across extern C when std panic catching is available. Return BrotliFileNotCraftedForConcatenation on caught panics and keep the existing pass-through behavior for no-std or pass-through-ffi-panics builds. Add regression coverage for a crafted stream input that previously panicked through BroccoliConcatStream. Reject serialized BroCatli buffers with out-of-range live state fields before constructing the state. This keeps deserialize_from_buffer on its existing Result<BroCatli, ()> API while returning Err(()) for corrupt buffers that would otherwise panic on later use

What's new in 8.0.3

Fix: avoid panic across Broccoli FFI boundary with BroCatLi Fix: CompressMulti worker joins on errors

What's new in 8.0.2

Fix for memory leak in ffi API

What's new in 8.0.1

Compatibility for ffi builds

What's new in 8.0.0

• Fixed LZ77 to comply with the specification
  • No longer seed the context by the end of the lz77 dictionary. Instead use zero for the seed as would happen without a dictionary. This matches the behavior of brotli with a custom dictionary as specified in
  • https://datatracker.ietf.org/doc/draft-vandevenne-shared-brotli-format/

What's new in 7.0.0

• Fix error with short writes
• allow quality=10 for certain APIs and make it default to 9.5

What's new in 6.0.0

• Remove unused SIMD use statements
• hide a few warnings - these are TODOs, and should be fixed in separate PRs
• do NOT build SIMD as part of MSRV -- doesn't make any sense to combine nightly with MSRV

What's new in 5.0.0

• The FFI is no longer active by default to avoid ODR issues if multiple versions of brotli are included in several dependent crates.

What's new in 4.0.0

Pinned to a rust-brotli-decompressor that can disable the ffi with the ffi-api flag. This can help avoid symbol conflicts with other brotli libs.

What's new in 3.5

Updated SIMD support. Better CI integration. Cleaned up some of the clippy warnings.

What's new in 3.4

Brotli decompressor's reader and writer has better behavior when operating upon brotli streams with extra bits at the end. Optional features like stdsimd are now tested or disabled for now.

What's new in 3.2

• into_inner conversions for both Reader and Writer classes

What's new in 3.0

• A fully compatible FFI for drop-in compatibiltiy with the https://github.com/google/brotli binaries
  • custom allocators fully supported
• Multithreaded compression so multiple threads can operate in unison on a single file
• Concatenatability mode to add the feature requested in google/brotli#628
  • binary tool catbrotli can accomplish this if the first file was specified with -apendable and the second with -catable
• validation mode where a file is double-checked to be able to be decompressed with the same settings; useful for benchmarking or fuzzing
• Magic Number: where the brotli file can have a useful header with a few magic bytes, concatability info and a final output size for pre-allocating memory

What's new in 2.5

• In 2.5 The callback also passes down an allocator to make new StaticCommands and PDFs and 256 bit floating point vectors.
• In 2.4 The callback with the compression intermediate representation now passes a full metablock at a time. Also these items are mutable in case futher optimization is desired

What's new in 2.3

• Flush now produces output instead of calling finish on the stream. This allows you to use the writer abstraction to get immediate output without having to resort to the CompressStream internal abstraction

Project Requirements

Direct no-stdlib port of the C brotli compressor to Rust

no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.

This is useful to see how C and Rust compare in an apples-to-apples comparison where the same algorithms and data structures and optimizations are employed.

Compression Usage

Rust brotli currently supports compression levels 0 - 11 They should be bitwise identical to the brotli C compression engine at compression levels 0-9 Recommended lg_window_size is between 20 and 22

With the io::Read abstraction

let mut input = brotli::CompressorReader::new(&mut io::stdin(), 4096 /* buffer size */,
                                              quality as u32, lg_window_size as u32);

then you can simply read input as you would any other io::Read class

With the io::Write abstraction

let mut writer = brotli::Compressor::new(&mut io::stdout(), 4096 /* buffer size */,
                                         quality as u32, lg_window_size as u32);

There are also methods to build Compressor Readers or Writers using the with_params static function

eg:

let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::Compressor::with_params(&mut io::stdout(), 4096 /* buffer size */,
                                         params);

or for the reader

let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::CompressorReader::with_params(&mut io::stdin(), 4096 /* buffer size */,
                                                       params);

With the Stream Copy abstraction

match brotli::BrotliCompress(&mut io::stdin(), &mut io::stdout(), &brotli_encoder_params) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

Decompression Usage

With the io::Read abstraction

let mut input = brotli::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);

then you can simply read input as you would any other io::Read class

With the io::Write abstraction

let mut writer = brotli::DecompressorWriter::new(&mut io::stdout(), 4096 /* buffer size */);

With the Stream Copy abstraction

match brotli::BrotliDecompress(&mut io::stdin(), &mut io::stdout()) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

With a custom dictionary

A raw LZ77 prefix dictionary is attached before the first read (or the first write, for DecompressorWriter); up to 15 may be attached. The dictionary is handed over as the allocator's memory type, and attach_dictionary returns false if it could not be attached.

use brotli::enc::StandardAlloc;
use brotli::{Allocator, Decompressor, SliceWrapperMut};

let mut alloc = StandardAlloc::default();
let mut dict = <StandardAlloc as Allocator<u8>>::alloc_cell(&mut alloc, dictionary.len());
dict.slice_mut().clone_from_slice(dictionary);

let mut input = brotli::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);
assert!(input.attach_dictionary(dict));

The dictionary lives in its own buffer rather than in the ring buffer, so it stays addressable for the whole stream even when it is larger than the window or when the output outgrows the window.

attach_serialized_dictionary takes the same kind of argument, but the bytes are a serialized shared dictionary container (magic 0x91 0x00) which may carry an LZ77 prefix dictionary, a custom word list and a custom transform list.

The matching binary tool flag is -customdictionary=<file>, on both compression and decompression:

brotli -c -customdictionary=dict.bin < plaintext > compressed.br
brotli -customdictionary=dict.bin < compressed.br > plaintext

With manual memory management

There are 3 steps to using brotli without stdlib

1. setup the memory manager
2. setup the BrotliState
3. in a loop, call BrotliDecompressStream

in Detail

// at global scope declare a MemPool type -- in this case we'll choose the heap to
// avoid unsafe code, and avoid restrictions of the stack size

declare_stack_allocator_struct!(MemPool, heap);

// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
let mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);
let mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);
let mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
let heap_u8_allocator = HeapPrealloc::<u8>::new_allocator(4096, &mut u8_buffer, bzero);
let heap_u32_allocator = HeapPrealloc::<u32>::new_allocator(4096, &mut u32_buffer, bzero);
let heap_hc_allocator = HeapPrealloc::<HuffmanCode>::new_allocator(4096, &mut hc_buffer, bzero);

// At this point no more syscalls are going to be needed since everything can come from the allocators.

// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.

// Now it's possible to setup the decompressor state
let mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);

// at this point the decompressor simply needs an input and output buffer and the ability to track
// the available data left in each buffer
loop {
    result = BrotliDecompressStream(&mut available_in, &mut input_offset, &input.slice(),
                                    &mut available_out, &mut output_offset, &mut output.slice_mut(),
                                    &mut written, &mut brotli_state);

    // just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
}

This interface is the same interface that the C brotli decompressor uses

Also feel free to use custom allocators that invoke Box directly. This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation

Using the C interface

rust-brotli is a drop-in replacement for the official https://github.com/google/brotli C implementation. That means you can use it from any place that supports that library. To build rust-brotli in this manner enter the c subdirectory and run make there

cd c && make

this should build c/target/release/libbrotli.so and should build the vanilla command line tool in C for compressing and decompressing any brotli file.

the libbrotli.so in c/target/release should be able to replace any other libbrotli.so file, but with all the advantages of using safe rust (except in the FFI bindings)

The code also allows a wider range of options, including forcing the prediction mode (eg UTF8 vs signed vs MSB vs LSB) and changing the weight of the literal cost from 540 to other values.

Custom dictionaries are reachable from C as well: brotli/decode.h declares BrotliDecoderAttachDictionary, which takes a BrotliSharedDictionaryType of either BROTLI_SHARED_DICTIONARY_RAW or BROTLI_SHARED_DICTIONARY_SERIALIZED from brotli/shared_dictionary.h. As in the C library, the dictionary must be attached before decoding starts, ownership is not transferred, and the buffer must stay alive until the decoder instance is destroyed.

BrotliDecoderState *state = BrotliDecoderCreateInstance(NULL, NULL, NULL);
if (!BrotliDecoderAttachDictionary(state, BROTLI_SHARED_DICTIONARY_RAW,
                                   dict_size, dict)) {
  /* dictionary was corrupt, or the 15 dictionary limit was reached */
}

Stream Concatenation

Brotli supports creating streams that can be concatenated together, useful for streaming scenarios where you want to compress chunks independently but decompress as a single stream.

Simple Concatenation (Fast)

Use -bare -appendable for the first file and -bare -catable for subsequent files. These can be combined using plain byte concatenation without special tools, with a finalization byte (0x03) added at the end:

# Create the base file with header but no trailer (must specify window size)
brotli -c -bare -appendable -w22 input1.txt > base.br

# Create bare-catable streams (no header, no trailer, same window size!)
brotli -c -bare -catable -w22 input2.txt > part2.br
brotli -c -bare -catable -w22 input3.txt > part3.br

# Simple concatenation with finalization byte
# Note: printf '\x03' adds the required final byte
(cat base.br part2.br part3.br; printf '\x03') > combined.br

# Decompress normally
brotli -d combined.br -o output.txt

Advantages:

• Instant concatenation (no processing)
• No special tools required
• Bare streams can be appended in any order

Requirements:

• All files must use the same window size (-w22 recommended)
• First file: -bare -appendable (has header, no trailer)
• Subsequent files: -bare -catable (no header, no trailer, no dictionary refs)
• A final 0x03 byte must be appended to complete the stream

Efficient Concatenation (Size-optimized)

Use the catbrotli tool with -catable and -appendable flags for better compression at the cost of processing time:

# Create files for catbrotli tool
brotli -c -appendable input1.txt > appendable.br
brotli -c -catable input2.txt > catable1.br
brotli -c -catable input3.txt > catable2.br

# Concatenate using catbrotli tool
catbrotli appendable.br catable1.br catable2.br > combined.br

Tradeoff: catbrotli produces smaller output but requires CPU time to process the streams intelligently. Use this when size matters more than concatenation speed.

Technical Reference: Stream Parameter Interactions

Stream Types and Their Parameters:

Stream Type	bare_stream	byte_align	appendable	catable	use_dictionary	Description
Standard	false	false	false	false	true	Normal brotli stream with header and trailer
First (simple concat)	true	true	true	false	true	Has header, no trailer - for simple cat concatenation
Subsequent (simple concat)	true	true	true	true	false	No header, no trailer, no dict refs - append to first
Appendable (catbrotli)	false	varies	true	false	true	For use with catbrotli tool
Catable (catbrotli)	false	varies	true	true	false	For use with catbrotli tool

Important Notes:

• Parameter dependencies are applied automatically by the library in both CLI and API usage
• The library's SanitizeParams function ensures:
  • catable = true → automatically sets appendable = true and use_dictionary = false
  • bare_stream = true → automatically sets byte_align = true
  • !appendable → automatically sets byte_align = false
• When using set_parameter(), dependencies are applied immediately
• When setting fields directly (e.g., params.catable = true), dependencies are applied during compression initialization
• No manual fixups needed - the library handles all parameter dependencies
• The use_dictionary = false for catable streams prevents references to bytes before the chunk boundary
• Simple concatenation requires a final 0x03 byte to complete the stream
• All concatenated streams must use the same window size

Example API Usage:

// First file: -bare -appendable equivalent
params.bare_stream = true;    // Sets bare_stream=true, byte_align=true (automatic)
params.appendable = true;     // Sets appendable=true

// Subsequent files: -bare -catable equivalent
params.bare_stream = true;    // Sets bare_stream=true, byte_align=true (automatic)
params.catable = true;        // Sets catable=true, appendable=true, use_dictionary=false (automatic)

// All parameter dependencies are handled automatically by the library.
// No manual fixups required - just set the primary flags you want.