TL:DR;

Integrating file uploads and downloads across different systems poses challenges due to varying requirements and constraints. We tackled these by:

1. Unified Upload Approach: Supporting both form data and binary uploads with dynamic header mapping.

2. Handling Data Requirements: Setting specific headers for Azure DevOps and Zendesk.

3. Two-Step Upload Process: Uploading files first, then linking them to entities.

4. Helpdesk Constraints: Sending attachments as a message in the ticket.

5. Download Management: Handling direct secured links and complex redirect URLs.

6. Content-Type Inconsistencies: Implementing a MIME type mapping mechanism for consistent handling.

By addressing these issues, we created a robust solution for seamless file upload and download integrations.

Introduction

In the world of software integrations, handling file uploads and downloads can be particularly challenging. This is especially true when dealing with multiple systems, each with its own requirements and quirks. In this post, we'll walk you through the issues we've encountered while adding support for uploading and downloading attachments and how we addressed them. We'll cover handling different file upload formats, managing various data requirements, working with specific constraints of helpdesk connectors, handling direct secured links, managing redirect URLs, and dealing with MIME type inconsistencies.

The Upload Conundrum

When dealing with multiple file upload formats, two common approaches often come up:

1. Form Data Uploads: This method involves sending files as part of a form data. It’s widely used and supported by most APIs.

2. Binary Uploads: In this approach, files are uploaded as raw binary data. This can be more efficient but is less commonly supported and requires specific handling.

After experimenting with both formats, we realised that a unified approach to file uploads was essential. The goal was to abstract the differences and provide a seamless experience regardless of the underlying API.

Data Requirements: One Size Does Not Fit All

Each integration has its own data requirements. Here are two notable examples:

1. Azure DevOps (ADO): ADO mandates that the Content-Type header for file uploads must always be set to application/octet-stream.

2. Zendesk: Zendesk, on the other hand, requires the Content-Type header to reflect the file’s actual MIME type.

To address these differences, we developed a dynamic header-mapping mechanism. This mechanism maps the headers as required by the underlying API and adjusts the headers accordingly.

The Two-Step Upload Process

Another major issue is that many integrations require a two-step process for attaching files:

Without Truto, you would need to take the following approach -

Upload the Attachment: First, you upload the file to the system's shared storage via their API. This returns an identifier or URL for the uploaded file.

curl "https://{subdomain}.zendesk.com/api/v2/uploads?filename=user_crash.log&token={optional_token}" \\ 
  --data-binary @crash.log \\ 
  -H "Content-Type: text/plain" \\ 
  -v -u {email_address}/token:{api_token} \\ 
  -X POST

Link the Attachment: Next, you use another API call to link the uploaded file to a specific entity, such as a ticket or comment.

curl <https://example.zendesk.com/api/v2/tickets/45135> \\ 
  -d '{"ticket": {"comment": {"body": "Press play", "uploads": ["4bLLKSOU63CPqaIeOMXYyXzUh"]}}}' \\ 
  -H "Content-Type: application/json" \\ 
  -v -u {email_address}/token:{api_token} -X PUT

With Truto Unified API, you only need to make a single API call -

curl --location '<https://api.truto.one/unified/ticketing/attachments?integrated_account_id=8f110dbe-0930-440f-b355-6589336ff4fa>' \\ 
  --header 'Authorization: Bearer {truto_api_token}' \\ 
  --form 'file=@"/Users/uday/Desktop/Screenshot 2023-12-26 at 4.28.12 PM.jpg"' \\ 
  --form 'ticket[id]="26"'

The above cURL command works with other integrations as well, as long as the required body fields are added.

Helpdesk Connectors: A Special Case

For helpdesk systems, a common constraint is that files can only be attached when a ticket is created. If you need to upload or link a new file to an existing ticket, you must send it as a separate message. This can be a bit tricky to manage, but we made it a unified way by accepting ticket[id] as a body field in the same form data and handling the mapping at Truto’s end so that there won’t be any additional implementation needed.

The Download Dilemma

Various integrations provide different ways to download an attachment or file. Here are the main scenarios we've encountered:

1. Direct Secured Links: These are the easiest to handle. The integration provides a direct link to the file, often secured with authentication tokens or headers. Downloading from these links is straightforward and efficient.

2. Redirect URLs: Some integrations provide URLs that redirect to the secured link. This introduces additional complexity, especially when dealing with authentication and headers.

Managing Redirect URLs

Redirect URLs can be particularly tricky. Here’s what we’ve found:

1. Simple Redirects: Some redirect URLs work seamlessly even if the original authentication headers are present. These are relatively easy to manage.

2. Strict Redirects: Other redirect URLs have strict checks and do not accept any headers during the redirection. In such cases, we need to manually scrape and handle the headers while making the follow-up request. Below is an example of how you can achieve this using the Fetch API -

Initial Request to Get Redirect URL:

async function fetchRedirectUrl(url, headers) { 
  const response = await fetch(url, { 
    method: 'GET', 
    headers: headers, 
    redirect: 'manual' // Prevent automatic redirection 
  }); 
  
  if (response.status === 302) { 
      const redirectUrl = response.headers.get('Location'); 
      return redirectUrl; 
  } else { 
      throw new Error('Failed to get redirect URL'); 
  } 
}

Follow-up Request to the Redirect URL:

async function fetchFile(redirectUrl) { 
  const response = await fetch(redirectUrl, { 
    method: 'GET', // No headers included here due to strict redirect policy 
    }); 
  
  if (!response.ok) { 
    throw new Error('Failed to fetch the file'); 
  } 
  
  const fileData = await response.blob(); 
  return fileData; 
}

With Truto Unified Download API, you need not worry about any of this and can download the file with single API call as shown below -

curl --location --request POST '<https://api.truto.one/unified/ticketing/attachments/download?integrated_account_id=51a9b9b6-2df1-4f70-bbe3-a32e819121f0&file_url=https%3A%2F%2Fprod-apse2-files-monday-com.s3.ap-southeast-2.amazonaws.com%2F24585066%2Fresources%2F12170011%2Fmiro-logo.png%3Fresponse-content-disposition%3Dattachment%26X-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIA4MPVJMFXILAOBJXD%252F20240724%252Fap-southeast-2%252Fs3%252Faws4_request%26X-Amz-Date%3D20240724T120622Z%26X-Amz-Expires%3D3600%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Signature%3D17df9d812436af5c66f44a63f0e968c4416cc40c28ef8d99a043ed601c066724>' \\ 
  --header 'Authorization: Bearer {truto_api_token}'

Handling Content-Type Header Issues

Another major issue we face is the inconsistent handling of the Content-Type header by different integrations. Some integrations set the Content-Type header to the correct MIME type, while others default to application/octet-stream. This inconsistency poses a challenge in keeping track of the correct MIME type and mapping the response header accordingly.

To address this, we implemented a MIME type mapping mechanism. We set the MIME type to our Unified Download URL and download the file. Once the API call is made successfully, we map the Content-Type header to the MIME type which we get from the query and send it as a response.

Summing it up

Integrating file uploads and downloads across different systems can be challenging due to varying requirements and constraints. By developing a unified approach to handle multiple file upload formats, dynamically setting headers, managing the two-step upload process, handling direct and redirect links, managing strict header requirements, and mapping MIME types dynamically, we were able to create a robust solution. Understanding and accommodating the specific needs of each system was key to overcoming these challenges.