org.xerial.snappy

Class SnappyOutputStream

java.lang.Object

java.io.OutputStream

org.xerial.snappy.SnappyOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

Direct Known Subclasses:

SnappyHadoopCompatibleOutputStream

public class SnappyOutputStream
extends OutputStream

This class implements a stream filter for writing compressed data using Snappy.

The input data is blocked into 32kb size (in default), and each block is compressed and then passed to the given OutputStream.

The output data format is:

1. snappy codec header defined in SnappyCodec (8 bytes)
2. codec version (4 bytes integer, at least 1) and compatible version (4 bytes integer)
3. compressed block 1 : a pair of (compressed data size [4 byte integer. Big-endian], compressed data...)
4. compressed block 2
5. ...

Note that the compressed data created by SnappyOutputStream cannot be uncompressed by Snappy.uncompress(byte[]) since the output formats of Snappy.compress(byte[]) and SnappyOutputStream are different. Use SnappyInputStream for uncompress the data generated by SnappyOutputStream.

Author:

leo

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCK_SIZE
protected byte[]	inputBuffer
static int	MAX_BLOCK_SIZE
static int	MIN_BLOCK_SIZE
protected OutputStream	out
protected byte[]	outputBuffer

Constructor Summary

Constructors

Constructor and Description
SnappyOutputStream(OutputStream out)
SnappyOutputStream(OutputStream out, int blockSize)
SnappyOutputStream(OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)

Method Summary

Modifier and Type	Method and Description
void	close() close the stream
protected void	compressInput()
protected void	dumpOutput()
void	flush()
void	rawWrite(Object array, int byteOffset, int byteLength) Compress the raw byte array data.
void	write(byte[] b, int byteOffset, int byteLength)
void	write(double[] f) Compress the input array data
void	write(double[] f, int off, int len) Compress the input double array data
void	write(float[] f) Compress the input array data
void	write(float[] f, int off, int len) Compress the input float array data
void	write(int b) Writes the specified byte to this output stream.
void	write(int[] f) Compress the input array data
void	write(int[] f, int off, int len) Compress the input int array data
void	write(long[] d) Compress the input array data
void	write(long[] d, int off, int len) Compress the input long array data
void	write(short[] f) Compress the input array data
void	write(short[] f, int off, int len) Compress the input short array data
protected void	writeBlockPreemble() Optionally write a preemble before a block.
protected void	writeCurrentDataSize()
protected int	writeHeader()

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_BLOCK_SIZE

public static final int MAX_BLOCK_SIZE

See Also:

Constant Field Values

MIN_BLOCK_SIZE

public static final int MIN_BLOCK_SIZE

See Also:

Constant Field Values

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

out

protected final OutputStream out

inputBuffer

protected byte[] inputBuffer

outputBuffer

protected byte[] outputBuffer

Constructor Detail

SnappyOutputStream

public SnappyOutputStream(OutputStream out)

SnappyOutputStream

public SnappyOutputStream(OutputStream out,
                          int blockSize)

Parameters:

out -

blockSize - byte size of the internal buffer size

Throws:

IllegalArgumentException - when blockSize is larger than 512 MiB

SnappyOutputStream

public SnappyOutputStream(OutputStream out,
                          int blockSize,
                          BufferAllocatorFactory bufferAllocatorFactory)

Method Detail

write

public void write(byte[] b,
                  int byteOffset,
                  int byteLength)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

write

public void write(long[] d,
                  int off,
                  int len)
           throws IOException

Compress the input long array data

Parameters:

d - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(double[] f,
                  int off,
                  int len)
           throws IOException

Compress the input double array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(float[] f,
                  int off,
                  int len)
           throws IOException

Compress the input float array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(int[] f,
                  int off,
                  int len)
           throws IOException

Compress the input int array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(short[] f,
                  int off,
                  int len)
           throws IOException

Compress the input short array data

Parameters:

f - input array

off - offset in the array

len - the number of elements in the array to copy

Throws:

IOException

write

public void write(long[] d)
           throws IOException

Compress the input array data

Parameters:

d -

Throws:

IOException

write

public void write(double[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(float[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(int[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

write

public void write(short[] f)
           throws IOException

Compress the input array data

Parameters:

f -

Throws:

IOException

rawWrite

public void rawWrite(Object array,
                     int byteOffset,
                     int byteLength)
              throws IOException

Compress the raw byte array data.

Parameters:

array - array data of any type (e.g., byte[], float[], long[], ...)

byteOffset -

byteLength -

Throws:

IOException

write

public void write(int b)
           throws IOException

Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.

Specified by:

write in class OutputStream

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

dumpOutput

protected void dumpOutput()
                   throws IOException

Throws:

IOException

compressInput

protected void compressInput()
                      throws IOException

Throws:

IOException

writeHeader

protected int writeHeader()

writeBlockPreemble

protected void writeBlockPreemble()

Optionally write a preemble before a block. Hadoop requires the actual block data size being written. This base implementation does nothing. Derive implementation can call writeCurrentDataSize().

writeCurrentDataSize

protected void writeCurrentDataSize()

close

public void close()
           throws IOException

close the stream

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException