# Parallel Fetch

Wrapper over node-fetch which adds supports for parallel downloading of files with pause/resume support.

## Delta from node-fetch

- The initialization options for node-fetch have an added option.

```Typescript
{
    parallel?: {
        minimumPartialSize?: number; // Minimum size of data to create a new partial for. Defaults to 2 MB. If a socket is done fetching and more than 4MB is pending on a parallel request, a new HTTP request will be sent to speed up the download. The older workload of the older connection is trimmed to 50% of the remaining and the rest will be taken care of by this new connection. Note that making this smaller than the high Water Mark passed on to node-fetch can cause a waste of resources due to data pulled into a buffer but requested again.
        maxSockets?: number; // Maximum number of parallel requests to send to the server. Defaults to 1
        maxLatency ?: number; // If some request has been stuck for a long time (node-fetch hung or bad socket or some server with a noisy neighbor), that request can be aborted and restarted. Max latency defines the time on when to give up. Value in milliseconds. Default 500ms.
        maxLatencyRetries?: number; // Maximum number of retries to do due to latency timeouts. Default  5.
    },
    timeout?: number; // Timeout like node-fetch v2. There is a difference in implementation. While the node-fetch v2 timeout was valid for the entire response duration, this timeout is valid till only the original response, after which the maxLatency is used.
}
```

- If parallel is passed as an option, the buffer, json and text properties of response are modified to provide parallel response handling.
- If an inner partial request gets a error, that error is thrown. Non OK responses for partial content are treated as errors.
- A new method load is provided to load a partial downloaded file which returns options to continue downloading.

- New options have been added to the response object:

* `file` A file is downloaded on disk. The path to the file needs to be provided as a parameter.
* `emitter` Provides events for manual data handling . Since the data is not available in sequence, the offset is passed in the data parameter. Note that a special progress event is also emitted by the emitter object that can be used to track download progress. It provides the number of bytes completed and the total number of bytes in the system. The total is not provided in case the content is compressed.

- If Content-Encoding is present and is no identity, the server uses compression. In this case, parallel download and resume is not supported. For images/binary data, it is not recommended to use compression. You can pass Accept-Encoding as identity to disable compression or pass compress: false to node-fetch to handle decompression manually. See https://github.com/node-fetch/node-fetch/issues/384

## Pause-Resume

Each downloaded file in progress is appended with a 20KB metadata block which contains the current state of the download and the parameters required to resume it. This includes the URL, the content length, the number of connections, progress details about each connection. This information is parsed and loaded on to RAM when the resume happens and can be read from the response object retrieved from the loading. The response object also contains a method called resume that can be used to resume a download. All initialization options, including the url can be overridden in the resume method. This allows support for handling cases where the URL or headers change across downloads.

The pause resume as well as parallel downloads use the If-Unmodified-Since and the ETag headers provided in HTTP to make sure that the content continues to be the same. A rejection is done if the content has been modified on the server during the download and the old content is no longer accessible.
