Batch Processor Documentation

Purpose

Provide a means to process a file of many address records through the LightBox goecoder at the same time. Batch processing is significantly more efficient due to the dramatic reduction in network latency caused by individual geocoding requests. Our recommendation is to use Batch Processing for workloads outside of interactive geocoding or autocomplete use cases.  Multiple files can be uploaded and processed simultaneously.  Each file or job can have a unique or consistent configuration of the data attributes to retrieved.  Batch processes are secured as they are only accessible to the user creating the uniquely identified job, each job has a private job identifier.  Batch jobs can be started, canceled and checked for the current process status including an estimated time to complete

Features

• Obtain a secure upload and download link for the CSV files 

• A secure job token, used to monitor the job with secure access to the job’s files

• The ability to start or cancel a job

• Supports very large files > 5GB  

• Flexible field output

• Job processing status with estimated time to complete

The process involves:

• Obtaining a secure and unique upload URL and a job token to a secure cloud storage

• Uploading the CSV file to a specific job token

• Start the geocoding process

• Use the job token to monitor the job

• From the job token, retrieve a secure download link to the completed CSV file

• Download the CSV file

API Requests

File Upload < 5GB

Use this endpoint to retrieve an upload URL, to be used to upload your CSV file to a secure location. 

POST /batch/files/upload

Example requests

curl -d '{"fileName": "file.csv"}' -H "Content-Type: application/json" -X POST http://api.lightboxre.com/v1/batch/files/upload

Request Body

{
  "fileName": "file.csv"
}

Response

Media type: application/json

{
  "$ref": "string",
  "signedUrl": {
    "expiry": "60 minutes",
    "token": "3b4ca2ae-4d64-46a3-b08b-2d088682f78c",
    "url": "https://foo.amazonaws.com/uat/files/5555555-4d64-46a3-b08b-2d088682f78c/filename.csv?X-Amz-Algorithm=AWS4-5555555SHA256&X-Amz-Credential=555555555PFKBAAUEOSV3%2F20240731%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240731T181736Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=PutObject&X-Amz-Signature=55555555555c8f7d61848317ba3a80863eb3d2310c74467edad295928"
  }

File Upload > 5GB

Use this endpoint to retrieve a multipart file upload URL for files that are greater than 5GB in size, which is used to upload large CSV files to a secure location. The total set of files is considered for a single batch processing job. Each file must be a well-formed CSV file.

POST /batch/files/initiate-upload

Example requests

curl -d '{"fileName": "file.csv", "numberOfParts": 5}' -H "Content-Type: application/json" -X POST http://api.lightboxre.com/v1/batch/files/initiate-upload

Request Body

{
  "fileName": "file.csv",
  "numberOfParts": 5
}

Response

Media type: application/json

{
  "$ref": "string",
  "signedUrl": {
    "expiry": "60 minutes",
    "token": "3b4ca2ae-4d64-46a3-b08b-2d088682f78c",
    "urls": [
      "https://foo.amazonaws.com/uat/files/5555555-4d64-46a3-b08b-2d088682f78c/filename.csv?X-Amz-Algorithm=AWS4-5555555SHA256&X-Amz-Credential=555555555PFKBAAUEOSV3%2F20240731%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240731T181736Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=PutObject&X-Amz-Signature=55555555555c8f7d61848317ba3a80863eb3d2310c74467edad295928"
    ],
    "uploadId": "ef.KCNxfpJwo1ZUaDgUsN9W4aiYScrjmv1xNOYpSYaLmgh_UDguAo51N8FeNK4LS95NPZpEVoGtjaF0DMFS7u1goMFDkai3tDaCjUpsmBfRmSjRXSIXRwSB7jaxqHNef"
  }
}

Complete Multi-part upload for files > 5GB

This call is made once all the parts of a large file has been uploaded.

POST /batch/files/complete-upload

Example requests

curl -d '{"parts": [{"partnumber": 1, "eTag":"555"}],"token": "555", "uploadID": "555"}' -H "Content-Type: application/json" -X POST http://api.lightboxre.com/v1/batch/files/complete-upload

Request Body

{
  "parts": [
    {
      "partNumber": 2,
      "eTag": "KCNxfpJwo1ZUaDgUsN9W4aiYScrjmv1xNOYpSYa"
    }
  ],
  "token": "3b4ca555-4d64-46a3-b08b-2d088682f78c",
  "uploadID": "ef.555o1ZUaDgUsN9W4aiYScrjmv1xNOYpSYaLmgh_UDguAo51N8FeNK4LS95NPZpEVoGtjaF0DMFS7u1goMFDkai3tDaCjUpsmBfRmSjRXSIXRwSB7jaxqHNef"
}

Response

Media type: 204 No Content

Download URL 

This endpoint will provide a download URL for the completed file. 

GET /batch/files/download/{token}

Example requests

curl -X GET -H ‘x-api-key: (api_key)’ https://api.lightboxre.com/v1/batch/files/download/3b4ca555-4d64-46a3-b08b-2d088682f78c

https://api.lightboxre.com/v1/batch/files/download/3b4ca555-4d64-46a3-b08b-2d088682f78c

Parameters

Response

Media type: application/json

{
  "$ref": "string",
  "signedUrl": {
    "expiry": "60 minutes",
    "token": "3b4ca2ae-4d64-46a3-b08b-2d088682f78c",
    "url": "https://foo.amazonaws.com/uat/files/5555555-4d64-46a3-b08b-2d088682f78c/filename.csv?X-Amz-Algorithm=AWS4-5555555SHA256&X-Amz-Credential=555555555PFKBAAUEOSV3%2F20240731%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240731T181736Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&x-id=PutObject&X-Amz-Signature=55555555555c8f7d61848317ba3a80863eb3d2310c74467edad295928"
  }

Clean up files 

This endpoint provides the means to remove data related to a job from the system. 

DELETE /batch/files/{token}

Allows direct removal of the files, input, and output associated with the Batch Processing Job. 

Example requests

curl -X DELETE -H ‘x-api-key: (api_key)’ https://api.lightboxre.com/v1/batch/files/3b4ca555-4d64-46a3-b08b-2d088682f78c

https://api.lightboxre.com/v1/batch/files/3b4ca555-4d64-46a3-b08b-2d088682f78c

Parameters

Response

HTTP Code 202, Accepted, No Content returned, This status is returned if the job is currently running. The system will first cancel the job then remove the files.

HTTP Code 204, Successful, No Content returned

Start a Job 

Once the data has been uploaded and you have a job token, this endpoint will add the job to the queue for processing. 

POST /batch/process/token

Example requests

curl -d '{see below}' -H "Content-Type: application/json" -X POST http://api.lightboxre.com/v1/batch/process/{token}

Parameters

Request Body

{
  "apis": [
    {
      "name": "geocoding-engine",
      "url": "/v1/search?text={1}+{2}+{3}+{4}",
      "outputColumns": [
        {
          "source": "features.0.geometry.coordinates.0",
          "target": "latitude"
        }
      ]
    }
  ]
}

Response

Media type: application/json

{
  "$ref": "string",
  "status": {
    "fileName": "file.csv",
    "jobDescription": {
      "apis": [
        {
          "name": "geocoding-engine",
          "url": "/v1/search?text={1}+{2}+{3}+{4}",
          "outputColumns": [
            {
              "source": "features.0.geometry.coordinates.0",
              "target": "latitude"
            }
          ]
        }
      ],
      "rowCount": 5003
    },
    "performance": {
      "averageSpeed": 3.6,
      "currentSpeed": 3.6,
      "etc": "01 23:59:59",
      "processedPercentage": 0.5,
      "processedRecords": 3456,
      "startedAt": "2024-07-24T10:23:54.430884195-04:00",
      "updatedAt": "2024-07-24T10:23:54.430884195-04:00",
      "totalRecords": 3456
    },
    "status": "STARTING"
  }
}

Return status on a Job/Process 

This endpoint will provide status on a job or process. 

GET /batch/process/{token}

Example requests

curl -X GET -H ‘x-api-key: (api_key)’ https://api.lightboxre.com/v1/batch/process/3b4ca555-4d64-46a3-b08b-2d088682f78c

https://api.lightboxre.com/v1/batch/process/3b4ca555-4d64-46a3-b08b-2d088682f78c

Parameters

Response

Media type: application/json

{
  "$ref": "string",
  "status": {
    "fileName": "file.csv",
    "jobDescription": {
      "apis": [
        {
          "name": "geocoding-engine",
          "url": "/v1/search?text={1}+{2}+{3}+{4}",
          "outputColumns": [
            {
              "source": "features.0.geometry.coordinates.0",
              "target": "latitude"
            }
          ]
        }
      ],
      "rowCount": 5003
    },
    "performance": {
      "averageSpeed": 3.6,
      "currentSpeed": 3.6,
      "etc": "01 23:59:59",
      "processedPercentage": 0.5,
      "processedRecords": 3456,
      "startedAt": "2024-07-24T10:23:54.430884195-04:00",
      "updatedAt": "2024-07-24T10:23:54.430884195-04:00",
      "totalRecords": 3456
    },
    "status": "STARTING"
  }
}

Cancel a Job/Process 

This endpoint provides you with the means to cancel a job/process. 

DELETE /batch/process/{token}

Example requests

curl -X DELETE -H ‘x-api-key: (api_key)’ https://api.lightboxre.com/v1/batch/process/3b4ca555-4d64-46a3-b08b-2d088682f78c

https://api.lightboxre.com/v1/batch/process/3b4ca555-4d64-46a3-b08b-2d088682f78c

Parameters

Response

HTTP Code 202, Accepted, No Content returned

Geocoding Mapping Options (Source Output Columns)

This table outlines the mapping options from source to target and what each field represents. We recommend carrying the fields marked with a '*' in the Best Practice column as these fields are the most important. 

HTTP Error Codes