This payload extracts CSV format from the request and lets you define regex to control precise extraction.
With the Row and Column Index, you can now define how the column positions can be calculated. For example, consider a CSV input as provided in the following snippet, where the first column begins at the 0th field in the row, that are padded with commas until the next column, ex5 begins. This applies when the indexing is 0-based. If you choose to use 1-based indexing, the first column begins at 1 and subsequent fields are 2, 3, and so on. Based on these definitions, you can define the rule and its properties.
first, ex5, last, pick, ex6, city, ex1, ex2
John, wwww, Smith, mister, wwww, stamford, 333, 444
Adam, wwww, Martin, mister, wwww, fairfield, 333, 444
It is recommended to use the External Base64 encoding method in the Transform action type for the CSV codec. If the Standard Base64 method is used, then additional newline feeds are generated in the output.
The CSV implementation in the DSG does not support the following:
The fields for the CSV payload are as seen in the following figure.
If the CSV input includes NON-ASCII or Unicode data, then the Binary extract rule must be used before using the CSV extract rule.
If the CSV input file includes non-printable special characters, then to transform the data successfully, the user must add the csv-bytes-parsing parameter in the features.json file.
To add the parameter in the features.json file, perform the following steps.
- Login to the ESA Web UI.
- Navigate to Settings > System > Files.
- Open the features.json file for editing.
- Add the csv-bytes-parsing parameter in the features.json file. The csv-bytes-parsing parameter must be added in the following format:
{ "features": [ "csv-bytes-parsing" ] }
The properties for the CSV payload are explained in the following table.
Properties | Sub-Field | Description | Additional Information |
---|---|---|---|
Line Separator | Separator that defines where a new line begins. | ||
Skip Lines Matching Pattern | Regex pattern that defines the lines that need to be skipped. | For example, consider the following lines in the file: User, Admin, Full Access, Viewer Partial Access, User, Viewer, Admin No Access, Viewer, User, Root No Acess, Partial Access, Root, Admin
| |
Preserve Number of Columns | Select to check if the number of columns are equal to the column headers in a CSV file. If there is a mismatch between the actual number of columns and the number of column headers, then the rule stops processing further and an error appears in the log. If you clear this check box and a mismatch is detected, then the rule still continues to process the data. A warning appears in the log. | If the checkbox is selected, ensure that the data does not contain two or more consecutive Line Separators. For example, if the Line Separator is set to \n, the following syntax must be corrected.name, city, pin\n Joe, NY, 10\n Smith, LN, 12\n \n Remove the consecutive occurrences of \n | |
Row and Column Index | Select 0 if row and column counting begins at 0 or 1 if it begins at 1. | 0 | |
Header Line Number | Line number with column headers. |
| |
Data Starts at Line | Line number from which the data begins. | Value calculated as Header Line Number +1 | |
Column Separator | Value by which the columns are separated. | ||
Columns | List of columns to be extracted and for which values action is to be applied. For example, consider a .csv file with multiple columns such as SSN, Name, etc that need to be processed. | ||
Column Name/Index | Column Name or index number of the column that will be processed. For example, if the name of the 1 column is “Name”, the value in Column Name/Index would be either 1 or Name. For example, with Row and Column Index defined as 0, if the name of the 1st column is “Name”, the value in Column Name/Index would be either 0 or Name. | ||
Profile Name | Profile to be used to perform transform operations on the matched content. | ||
User Comments | Additional information related to the action performed by the column processing. | ||
Text Qualifier | Pattern that allows cells to be combined. | ||
Pattern | Pattern that applies to the cells, after the lines and columns have been separated. | ||
Advanced Settings | Define the quote handling for unbalanced quotes in CSV records.
| If quoteHandlingMode is set as DEFAULT, the unbalanced quotes are balanced. However, if the quote is followed by a string, the unbalanced quotes are not corrected by the DSG. For example, in the following CSV text, the quotes are not balanced by DSG: 'Joe,03/11/2024 or "Joe,13/11/2024 The output of this entry remains unchanged. |