Quick Start Guide - Rons Data Stream

1. Introduction

The aim of this Quick Start Guide is to help get started with Rons Data Stream. It requires a little time to get used to, however users who work regularly with data files will find this time investment worthwhile due to the amount of time saved at the end. Once the set up has been done, thousands of files can be processed in no time, in a few clicks on demand.

Two areas need to be set up to process files automatically: a Job and a Cleaner, which work together to process data.

Basically, the Cleaner represents the "What (happens)" and the Job the "To What".

In this guide, we are going to explain the basic flow and make it easy for our users to start using Rons Data Stream.

2. The Job

1. Purpose of a Job

The Job is the section where the source file(s), format(s) and output destination(s) are determined. They also allow file conversion between formats (Output Formaters settings), split files (Output Shapers settings) and produce a log file.

Jobs can be saved in File Explorer and re-used later.

To get started with creating a Job, click on New Job on the Start Page to open a new Job tab.

There are two predominant areas: the Job Palette and the Job Rules. The middle section displays the details of the selected Job Rules. In the right corner, an button opens a Preview Side Panel (see 4. The Preview).

Seven rules need to be set up with the necessary information to run the Job:

Source Containers
Source Profiles
Data Cleaner
Output Formats
Output Shapers
Output Containers
Output Loggers

Note: There are minimum settings required to to run a Job. If any are missing a warning message highlighted in yellow is displayed: This Rule cannot execute with the current settings (see picture).

2. The Job Rules

1. Source Containers

As described by the name, the Source Container Section contains the definition of entities that contain data for processing, in the form of files or data tables. The Palette contains these options for data containers:

Directory
Azure Blob Storage
SQL Server
Web Pages

The '1 - Directory Source Container' option is selected by default and we will use it as an example in this article. When clicking on it, it opens the settings in the middle of the screen. From there, the directory containing the files can be browsed. When a folder is selected, the files appear in the Job Output area after refreshing.

Note: Multiple sources or outputs can be selected allowing the processing of multiple files at the same time.

2. Source Profiles

Source Profiles define how the data in the Source Container is filtered and/or parsed. Multiple Profiles can use one Source Container.

Now that we have the directory Source Container, we need to choose the format of the files to be processed. The Source Profiles Palette displays a list of formats:

CSV (or any character Delimited format)
Token
XML
JSON
SQL Table
HTML Table

As an example, select CSV to add it to the Job Rules area, then click on the added 'Character Delimited Source' to open the details in the middle of the screen, and select the Source Container. In this case, because we only added one when we chose our Source container in the first step, only one will be displayed in the drop down menu. If we had added more in step 1, they would all be displayed.

3. Data Cleaners

Editing action(s), are defined by Cleaners that are linked to the Job settings. Cleaners can be shared between multiple Jobs and can be saved in File Explorer and re-used later.

In the Job section, this is the place where one or more Cleaner(s) can be added for the Job to be run. The Cleaner configuration has its own section in this guide (See 3. Data Cleaner).

What is important to know at this stage is that either an External (File) Cleaners or an Embedded Cleaner(s) can be selected from the Palette.

External (File) Cleaners: Cleaners can be saved and shared between Jobs, so this is where existing Cleaners can be added. Simply browse the path in the middle section to select the existing Cleaner(s). After a Cleaner was selected, it is possible to make changes to it. Clicking on Edit in the middle section. It will open the Cleaner in another tab where changes can be made and saved.

Embedded Cleaner(s): Selecting it means that a new Cleaner must be created and this can be done by clicking on Edit in the middle section. It will open the Cleaner in a new tab where changes can be made and saved.

Important tip: If you would like to check if changes the Cleaner(s) are making to the data are as expected, go back to the Job tab and open the Preview side panel. Each file can be previewed to shows the outcome of the editing action(s). Make sure to refresh that section regularly.

4. Output Formats

The Output Formats section is used to establish the format in which the file(s) output after processing.

Note: Rons Data Stream can simply be used as a great converter (with no Cleaner needed).

The format set up Palette selection is:

CSV
Token
HTML
XML
JSON
Text Template
SQL Table
SQLite Database

Several Output formats may be selected. For each selected format, the relating middle section setting area requires to choose an Output Container. That way the different formats can be saved into their own folders.

At this point, if the Output Containers weren't selected yet from the Palette, then they wouldn't show in the drop down menu in the Output Format setting area. Select them first from their Palette, then go back and then they will be listed in the drop down menu and it will be possible to select them. The Job wouldn't be able to be processed without them.

Important: In all settings, whenever a new name (or an identifier) needs to be created, one is always created automatically as it's a compulsory field. However, note that it can be customized and overwritten.

5. Output Shapers

The Output Shapers section allows to split the file(s) while they are processed. There are four different settings types that can be selected from the Palette:

Split - Row Count
Split - Row Ratio
Split - Row Filter
Split - Column Value

6. Output Containers

The Output Containers Section is the area used to determine where the processed files will be saved. In the Palette, three types of containers are listed:

Directory
Azure Blob Storage
SQL Server

7. Output Loggers

Processed files automatically generate a log file. In the Output Loggers section, one can select set up where the log file is to be saved by selecting the appropriate output container that has been created.

Because several Output Containers might have been created, then there might be different logfiles. Therefore the user might want to save the logfiles into different destinations. In this case, several logfiles can be selected from the Palette and then linked to each Output Containers that will be listed in the drop down menus. However, this is really a matter of preference and the logfile set up is optional.

3. The Cleaners

1. Purpose of a Cleaner

Now that we have the "Where and How" set up (The Job), we need the "What": the Cleaner.

A Cleaner defines the editing operations to be processed through the data files when running the Job. One or more cleaners may be saved and added to a Job setting in any order of execution.

After the Cleaners are created and saved, they can be integrated into the Data Cleaners Job rules section. If more than one Cleaner is integrated, the Job will process them in their visible order.

2. Cleaner Rules

To get Started with creating a Cleaner, click on New Cleaner on the Start Page: a new Cleaner tab is opened, with a Cleaner Palette section and the Cleaner Rules.

In the Cleaner Rules area, there are two Selectors (a Column and a Row Selector) to determine where the editing action should take place in the file, and three Processors sections (a Column, a Row and a Cell Processor) each listing specific editing operations.

Column Selectors
Row Selectors
Column Operations
Row Operations
Cell Operations

Each Operations can be selected from the Palette and then set up separately.

To add a new operation from the Palette, click on one of the Selectors or Processors from the Palette marked with a blue arrow to add it to the Cleaner Rules section. A colored number is added in front of it to help differ the various rules.

Note: Rules can be easily moved up and down, duplicated or deleted using the Home Menu options.

Click on the newly added rule to open the rule set up area in the right-side section and apply the desired set up.

All names can be edited to make the operations easily recognizable.

An incomplete operation that cannot execute is marked by a message with a hand sign: "This rule cannot execute with the current settings". It indicates that you need to look into the rule to make it functional.

1. Columns Selectors

Select the Columns where the editing will apply. There are three different selectors to be chosen from the Palette:

All Columns (Selected by default)
Column Range
Column List

This section is compulsory and must contain at list one rule.

2. Rows Selectors

Select the Rows where the editing will apply. There are five different Row Selectors that can be picked from the Palette:

Header Row (Selected by default)
All Rows (Selected by default)
Row Range
Empty/Filled Rows
Filter

Header Row and All Rows are selected by default which means that the editing rules will be applied all over the data sheet including the Header Row. However, even if they are selected by default for convenience, the rules can easily be deleted or deactivated by removing the click from the box if needed.

This section is compulsory and must contain at list one rule.

3. Column Operations

Editing actions that are specific to columns can be set up in this area. There are seven Column Processors that can be picked from the Palette:

Add Columns(s)
Update Column
Copy Column
Move Column
Delete Column
Merge Column
Split Column

4. Row Operations

There are three operations specific to rows that can be set up in this area.

Add Row
Delete Row
Split Row

5. Cell Operations

There are twenty-two Operations specific to cells can be set up in this area.

Add text
Add Expression Text
Add Random Text
Pad Text
Remove Text
Replace Text
Replace Lookup
Reverse Text
Format Text
Add Space
Remove Space
Add Numbers
Remove Numbers
replace Numbers
Add Date Time
Replace Date Time
Change Case
Split Name
Split URL
HTML Encode
HTML Extract
Fill Lookup

4. Finalizing the Job

The final steps happens in the Job.

When all the Jobs rules have been set up and include a Cleaner, the Job can be run by clicking on the Job Process button in the top menu or in the Job Options area of the Job Rules section.

However, before processing the Job, one might want to double-check if the actions affecting the files have been set up correctly and whether the outcome is going to be the one expected. To do so, the user can check the Job Preview.

Note: The Preview and the Process Job buttons are only active in a Job. In a Cleaner, the buttons are grayed out.

Job Preview

Clicking on the Preview button on the right corner of the screen opens a section that displays all the actions that are about to be processed. All the files to be edited are displayed and have a Preview button assigned to them.

Clicking on a file preview opens a window that displays the (Data) Source view and The (to be) Processed view. It also displays the Cell Change information (click on the Cell Change button): Whenever an area of the data is clicked on, the Column, Row or Cell operations involved are displayed with their number. This way the user can find out what operating rule is used for the editing operation.

If a change needs to be made, the user can open the relating Cleaner in a separate tab, make the change and immediately switch back to the Preview to see the new result. Make sure to click on Refresh!

Processing the Job

This is the easy part: when ready, simply click on the Press Job button. A progress section pops up briefly to display the tasks in Progress and whether they are still Processing, Complete, Pending or if there are Errors. The pop-up message will be displayed shortly but it can be re-opened anytime by clicking on View Tasks down on the right side corner in the below bar.

This area basically confirms if the Job processing has been successful, partially successful or failed.

5. Conclusion

Rons Data Stream is an awesome application that requires a little time to understand and to set up and we hope that this (possibly a little too long) Quick Start Guide will be helpful. However, that time is very well invested because, once the Jobs and the Cleaners created and combined, Rons Data Stream can run an incredible amount of (large) files or folders in no time and will be appreciated as a tremendous time saver.