Streaming files (Qlik Repository API)

Until now uploading and downloading files in Qlik Repository API package was done via buffers.

Generally this is (was) working well. The potential problem is when dealing with larger files. And as we know some of the Qlik Sense apps can reach quite dramatic sizes.

Buffers first fits all the data into memory and then write it locally (or upload it). And if the data, that is being read/written, is large then the process will consume a lot of resources.

Because of this Qlik Repository API will now download/upload files using streams. Streams read the data in chunks. Chunking the data make it more "digestible" aka less resources being used.

Code examples

Below are couple of examples how to download and upload apps.

Download app

// get instance of the app
const app = await repoApi.apps.get({
  id: "1111111-2222-3333-4444-555555555555",
});

// export the app request
// at this point "stream" variable will have "file" property
// "file" property will be of type IncomingStream
const stream = await app.export();

// prepare the writer. it will write the incoming data to the provided path
const writeToFile = fs.createWriteStream("location/to/write/some-app.qvf");

// Start piping the chunks of incoming data to the stream writer
stream.file.pipe(writeToFile);

// we have to wait for all the data to be received and written
// for this reason we'll "listen" for the "end" event on the stream writer
await new Promise((resolve, reject) => {
  stream.file.on("end", () => {
    resolve("");
  });

  stream.file.on("error", () => {
    reject();
  });
});

Upload app

const streamContent = fs.createReadStream("path/to/some-file.qvf");

const app = await repoApi.apps.upload({
  name: "Some app name",
  file: streamContent,
});

Bonus

A little bonus/side-effect of using streams is that now its possible to stream files between two (one source and at least one destination) Qlik Sense clusters.

This way the data, that is being downloaded and the data that is being uploaded, stays in memory!

The example below shows how to transfer an app from one Qlik Sense cluster to another:

// repoApi - source QS cluster. From where the app will be exported
// repoApiDestination - destination QS cluster. Where the app will be imported

// get instance of the app
const sourceApp = await repoApi.apps.get({
  id: "1111111-2222-3333-4444-555555555555",
});

// export the app request
// at this point "stream" variable will have "file" property
// "file" property will be of type IncomingStream
const streamSourceApp = await sourceApp.export();

// push the stream (chunked) data to the upload method
// "file" property accepts Buffer or IncomingMessage or createReadStream
// and streamSourceApp.file is of type IncomingMessage
const destinationApp = await repoApiDestination.apps.upload({
  name: "some name", // or if we want the same name - sourceApp.details.name
  file: streamSourceApp.file,
});