CSV Payload

Describes the CSV payload.

This payload extracts CSV format from the request and lets you define regex to control precise extraction.

With the Row and Column Index, you can now define how the column positions can be calculated. For example, consider a CSV input as provided in the following snippet, where the first column begins at the 0th field in the row, that are padded with commas until the next column, ex5 begins. This applies when the indexing is 0-based. If you choose to use 1-based indexing, the first column begins at 1 and subsequent fields are 2, 3, and so on. Based on these definitions, you can define the rule and its properties.

first, ex5, last, pick, ex6, city, ex1, ex2
John, wwww, Smith, mister, wwww, stamford, 333, 444
Adam, wwww, Martin, mister, wwww, fairfield, 333, 444

It is recommended to use the External Base64 encoding method in the Transform action type for the CSV codec. If the Standard Base64 method is used, then additional newline feeds are generated in the output.

The CSV implementation in the DSG does not support the following:

  • The fields contain line breaks.
  • If double quotes are used to enclose fields, then the double quotes appearing inside a field are escaped by preceding them with another double quote.

The fields for the CSV payload are as seen in the following figure.

CSV payload

If the CSV input includes NON-ASCII or Unicode data, then the Binary extract rule must be used before using the CSV extract rule.

If the CSV input file includes non-printable special characters, then to transform the data successfully, the user must add the csv-bytes-parsing parameter in the features.json file.

To add the parameter in the features.json file, perform the following steps.

  1. Login to the ESA Web UI.
  2. Navigate to Settings > System > Files.
  3. Open the features.json file for editing.
  4. Add the csv-bytes-parsing parameter in the features.json file. The csv-bytes-parsing parameter must be added in the following format:
    { "features": [ "csv-bytes-parsing" ] }

The properties for the CSV payload are explained in the following table.

PropertiesSub-FieldDescriptionAdditional Information
Line Separator Separator that defines where a new line begins. 
Skip Lines Matching Pattern Regex pattern that defines the lines that need to be skipped.For example, consider the following lines in the file: User, Admin, Full Access, Viewer Partial Access, User, Viewer, Admin No Access, Viewer, User, Root No Acess, Partial Access, Root, Admin
  • If you configure the regex as .*?User, the lines 1, 2, and 3 will be skipped.
  • If you configure the regex as User, the first line will be skipped and the remaining lines will be processed.
Preserve Number of Columns Select to check if the number of columns are equal to the column headers in a CSV file.
If there is a mismatch between the actual number of columns and the number of column headers, then the rule stops processing further and an error appears in the log.
If you clear this check box and a mismatch is detected, then the rule still continues to process the data. A warning appears in the log.
If the checkbox is selected, ensure that the data does not contain two or more consecutive Line Separators. For example, if the Line Separator is set to \n, the following syntax must be corrected.
name, city, pin\n Joe, NY, 10\n Smith, LN, 12\n \n
Remove the consecutive occurrences of \n
Row and Column Index Select 0 if row and column counting begins at 0 or 1 if it begins at 1.0
Header Line Number Line number with column headers.
  • -1 – Row and Column Index is 0
  • 0 – Row and Column Index is 1
Data Starts at Line Line number from which the data begins.Value calculated as Header Line Number +1
Column Separator Value by which the columns are separated. 
Columns List of columns to be extracted and for which values action is to be applied. For example, consider a .csv file with multiple columns such as SSN, Name, etc that need to be processed. 
 Column Name/IndexColumn Name or index number of the column that will be processed. For example, if the name of the 1 column is “Name”, the value in Column Name/Index would be either 1 or Name.
For example, with Row and Column Index defined as 0, if the name of the 1st column is “Name”, the value in Column Name/Index would be either 0 or Name.
 
 Profile NameProfile to be used to perform transform operations on the matched content. 
 User CommentsAdditional information related to the action performed by the column processing. 
Text Qualifier Pattern that allows cells to be combined. 
Pattern Pattern that applies to the cells, after the lines and columns have been separated. 
Advanced Settings Define the quote handling for unbalanced quotes in CSV records.
  • Set to {"quoteHandlingMode" : "DEFAULT"} to correct unbalanced quotes in records, such as, single quotes, in the delimited CSV input file during data processing.
    For example, if the CSV includes unbalanced quotes, such as, ',03/11/2020 or ",13/08/2020 and the Default Mode is enabled, then during data processing, the DSG will correct the unbalanced quotes. The DSG will change the unbalanced quotes to '',03/11/2020 or "",13/08/2020 respectively.
  • Set to {"quoteHandlingMode" : "PASSIVE"} to retain unbalanced quotes in records, such as single quotes, in the delimited CSV input file during data processing.
    For example, if the CSV includes a unbalanced quotes, such as, ',03/11/2020 or ",13/08/2020 and the Passive Mode is enabled, then during data processing, the DSG will retain the unbalanced quotes.
If quoteHandlingMode is set as DEFAULT, the unbalanced quotes are balanced. However, if the quote is followed by a string, the unbalanced quotes are not corrected by the DSG. For example, in the following CSV text, the quotes are not balanced by DSG: 'Joe,03/11/2024 or "Joe,13/11/2024 The output of this entry remains unchanged.
Last modified February 7, 2025