File Transfer

Overview

This page describes the transfer of flat files between the Oleeo ATS and Client system for integration between systems.

Integrations may provide new candidate application details from the ATS to Client and receive updates to previously communicated applications from Client to the ATS.

Integrations may receive vacancy information from a client system.

Information is passed by transferring files containing data in csv format, one created by the ATS for new applications, one created by Client for updates to applications, and one created by Client for Vacancies.

Files are transferred using an ip address restricted sftp service hosted by Oleeo.

Files are expected to be produced from the ATS and processed once per day but more frequent schedules of up to every 15 minutes can be considered.

Oleeo can support ‘one-way’ or ‘two-way’ integrations:

• One-way is where the receiving system would not send back any updates to the records received from the generating/sending system. (For example, Oleeo might send ‘new Hires’ to the Client at the end of the recruitment process, and then not receive any further updates from the client - the process for the application would be at a terminal status in the ATS. Alternatively. the Client may send details of new Vacancies to the ATS and not expect any updates back from the ATS in response.)
• Two-way is where the receiving system may create a response file as a result of processing new records from the sending system. The response may include error messages, status updates, or additional data elements. Currently Oleeo only expects two-way integrations for Applications sent from the ATS to the Client, and updates sent from the Client to the ATS.

File Format

Data files are CSV format according to RFC4180 https://www.ietf.org/rfc/rfc4180.txt. Text fields will always be enclosed in quotes. Text fields may include un-escaped carriage returns.

Oleeo default character encoding is UTF8

Default line ending is \r\n

Default column separation character is comma.

A data file contains one and only one row of data generated per source object (application, vacancy). All rows in a file contain the same number of columns.

Oleeo recommends that files are encrypted using pgp (Oleeo uses gpg Open Source utility for encryption). In this case, public PGP encryption key files must be exchanged between client and Oleeo.

After the data file is ready, an MD5 checksum of the final file (i.e. the encrypted contents where encryption is used) will be made available.

File Transfer Protocol

SFTP

Files may be transferred by sftp service hosted by Oleeo implemented using OpenSSH.

The service provides non-interactive authentication by way of authentication keys generated by ssh-keygen. The client should generate their own key pair and supply the public key to Oleeo. The client must then provide the identity file at the time of connection (install in appropriate directory or supply with the –i command line switch).

Use of ssh protocol version 2 is required.

Use of cipher aes256-gcm is recommended.

The service is available on Port 22 at sftp.tal.net but only to white-listed IP addresses. The client must supply the ip range used to access the service.

Files from Oleeo to the client are available in the directory /outgoing

The client should delete files from the outgoing directory after processing (Oleeo will assume that deletion is acknowledgement of successful processing by the Client). At any time, Oleeo may delete files older than 7 days.

The client can write files to the directory /incoming.

Oleeo picks up files in the /incoming directory and moves them to the /processed directory after processing is complete. The client can view the /processed directory to determine status of processing. At any time, Oleeo may delete files older than 7 days.

HTTPS

Oleeo recommend sftp but, where Oleeo provides the data, the files can be hosted on a web server in an username and password protected directory and available over https (tls v1.2).

Data from the ATS

Oleeo can support file transfer for different object types in the ATS. These object types include candidates, applications, and vacancies. By far the most common scenario is sending of application data for applications at the final stages of the hiring process.

Each data file can only include one object type. The data file contains one row for each instance of the object type matching the selection criteria. The selection is configured in the ATS as a ‘config saved search’ and can include any criteria that the standard ATS search can support. Typically this would be Applications at a particular Status like ‘Ready for Hire’, or that have a simple Flag (like ‘ExportStatus’) set to a value (like ‘Ready for Export’).

Oleeo ATS generates a data file containing details of instances of the object-type ready for downstream processing. Standard integration costs assume that no more than 50 columns are required in such files (more may be agreed, but may incur additional costs).

The data file is generated at a time and frequency agreed with the client.

Default filename is of the format: /outgoing/<systemID>_<objectType>_yyyymmddhhmmss.csv.pgp where:

• <systemID> is a numeric identifier unique to the Client’s ATS system. (This will remain constant for the life of the integration.)
• <objectType> is text description of the object that the file contains and is typically one of applications, candidates, vacancies.
• yyyy is current year, mm is current month, and dd is current day of month, hh is hour in 24 hour format, mm is minutes, ss is seconds; zero padding is applied where applicable e.g. 05 for month of May, 07 for hour of 7am.

The data file is encrypted using pgp.

After an instance of the object_type is added to the data file for export, the instrecord instance in the Oleeo ATS can have a Flag (like ‘ExportStatus’) can be set to a value (like ‘Exported’).

If an error is encountered while processing a matching application then a Flag (like ‘ExportStatus’) can be set to a value (like ‘Export Error’).

After a complete data file is generated and available on the sftp server, a manifest file is then created in the same directory. The manifest file contains an MD5 checksum of the data file on the sftp server (so is of the encrypted file if applicable). Please note, the use of MD5 is only intended for integrity checking, not for information security purposes; it does not present any sort of security threat. The manifest file has the same filename as the data file up to the first period but then has the extension md5 e.g. /outgoing/<systemID>_<objectType>_yyyymmddhhmmss.md5

If the MD5 file is not present then it should be assumed that processing is still on-going.

If the MD5 file content does not match the MD5 of the encrypted data file then this indicates a processing error.

If there is nothing to export at the time the export process runs then a file containing a header row but no content rows is produced together with a valid MD5 file.

Each row includes a unique object instance ID which can be used to reference the object when applying updates.

After the client has processed a file, then the Client should delete the files from the outgoing directory, including the manifest file.

Attachments

Attachments (files uploaded to the ATS) can be included in the file transfer integration. This is only expected for Candidate or Application object-types. This could be used to transfer Resume, Transcripts, or Identification documentation that might be required for a Hire record or for onboarding.

Attachments are not covered by the default Integration and may be an additional cost.

Where attachments are included then the CSV file will contain column(s) with values that correspond to a file name in an associated zip file. For each data file, there will be an additional zip file that constrains all attachments as referenced in the column data.

Default filename containing attachment is of the format: /outgoing/<systemID>_<objectType>_yyyymmddhhmmss.zip.pgp

Oleeo can enforce a limit on the total size of all attachments in any single file (the size is all attachments for all instances of object type before zip compression is applied).

An alternative to providing a separate zip file of attachments, attachments may be included within column values as base64 encoded text. This makes the file very large and difficult to read in a usual editor and is therefore more awkward to debug and not recommended.

Error Handling

If an error is encountered when exporting an application such that it cannot be processed (e.g. missing mandatory data) then the export status flag on the object_type instance in the ATS can be set accordingly (e.g “export issue”).

Any records missing mandatory fields are excluded from the file.

Two-way integration: Updates to applications in the ATS from clients

For two-way integration, the Client may provide a file back to the ATS to update the appropriate instances of the object-type in the ATS. Currently it is only expected that the object-type of applications would have a two-way integration.

The client generates a data file containing updates to applications that have previously been exported. This file can be used to indicate the status of processing of the application in the downstream system, and can update additional date elements (such as a Client generated ID). Standard integration costs assume that no more than 5 columns are required in such update files (more may be agreed, but may incur additional costs).

Oleeo monitors the /incoming directory and is notified when files are added. Processing occurs shortly after notification (i.e. processing is triggered by the upload event rather than scheduled to occur at a particular time or frequency).

Filename is of the format: /incoming/<systemID>_<objectType>_updates_yyyymmddhhmmss.csv.pgp where updates indicates that this is an update in response to an earlier transfer of instances of object_type

The data file is encrypted using pgp.

The data file contains one row for each object (application) to be updated. Objects (applications) to be updated are identified by the Oleeo Object ID (ApplicationID).

Once a complete data file has been uploaded, a manifest file must be created in the same directory. The manifest file contains an MD5 checksum of the encrypted data file. The manifest file has the same filename as the data file up to the first period but then has the extension md5 e.g. /incoming/<systemID>_<objectType>_updates_yyyymmddhhmmss.md5

If there are no updates then no file is uploaded and no manifest file is created.

After processing by Oleeo, files are moved to /processed directory. For each data file, Oleeo produces a status file containing the string “OK” (2 letters without the surrounding quotes) if the processing was successful. Any other content in the file indicates that an error was encountered.

The status file has the same filename as the data file up to the first period but then has the extension txt e.g. /processed/<systemID>_<objectType>_updates_yyyymmddhhmmss.txt

Inbound from client to ATS

The ATS can support inbound data from the client. This section described the inbound creation or update of instances of object_type sourced from a client system.

The majority of such integrations are for the import and update of Vacancies and the rest of this section describes that use-case only.

The client generates a data file containing new vacancies required to be imported into the Oleeo ATS.

Filename is of the format: /incoming/<systemID>_vacancies_yyyymmddhhmmss.csv.pgp

The data file is encrypted using pgp.

The data file contains one row for each new vacancy.

Once a complete data file has been uploaded, a manifest file must be created in the same directory. The manifest file contains an MD5 checksum of the delivered data file (including encryption if applicable). The manifest file has the same filename as the data file up to the first period but then has the extension md5 e.g. /incoming/<systemID>_vacancies_yyyymmddhhmmss.md5

If there are no updates then no file is uploaded and no manifest file is created.

Oleeo monitors the /incoming directory and is notified when files are added. Processing occurs shortly after notification (i.e. processing is triggered by the upload event rather than scheduled to occur at a particular time or frequency).

After processing by Oleeo, files are moved to /processed directory. For each data file, Oleeo produces a status file containing the string “OK” (2 letters without the surrounding quotes) if the processing was successful. Any other content in the file indicates that an error was encountered. These status files has the same filename as the data file up to the first period but then has the extension txt e.g. /processed/<systemID>_vacancies_yyyymmddhhmmss.txt

File contents - column definition

Some details of the file content, including the columns included in any file, can be configured according to the standard capabilities of the Oleeo ATS. The details of columns and values, and any changes to the default settings, must be approved by Oleeo. Oleeo can only use the standard configuration features of the ATS and so the file contents must be compatible with the standard ATS features. There are a lot of configurable options within the standard features including column formatting, value translation etc.