Advanced Dispatcher Configuration

This section details advanced dispatcher options configured in /opt/sonarfinder/sonarFinder/dispatcher.conf

RDBMS pull configration - see [Accessing SonarG Data Using SonarSQL](http://sonargdocs.jsonar.com/latest/sonarg_sql.html?highlight=rdbms “RDBMS”)

LDAP pull configuration - see : [SonarG Administration](http://sonargdocs.jsonar.com/latest/sonarg_admin.html?highlight=ldap “LDAP”)

Web Service Pull

Use the Web Service field to specify that you want a job to invoke a Web service and insert/update data within DCAP Central. The Web service response can either be in the form of JSON or CSVs, as specified in the section of dispatcher.conf that matches the alias entered in the Web Service field. When the response is a JSON document or array it is processed/parsed using SonarGateway. Therefore, when using this feature you must also add a config section to SonarGateway that accepts the data elements. If the response is a CSV then it is placed into sonargd/incoming and you can control its parsing through sonargd.conf and the misc section.

You can use multi-runs and generate multiple pulls as you would for standard API calls. For example, if you have a collection that contains:

{
  "_id": "sn2",
  "name": "ServiceNow2",
  "ticket": "INC0010002"
},
{
  "_id": "sn1",
  "name": "ServiceNow2",
  "ticket": "INC0010001"
}

you can enter a URL in the job named ServiceNow2:

  1. https://dev74625.service-now.com/incident.do?JSONv2&sysparm_action=getRecords&sysparm_query=number=”$$ticket”&displayvalue=true
causing the following two REST APIs to be called:
  1. https://dev74625.service-now.com/incident.do?JSONv2&sysparm_action=getRecords&sysparm_query=number=INC0010001&displayvalue=true
  2. https://dev74625.service-now.com/incident.do?JSONv2&sysparm_action=getRecords&sysparm_query=number=INC0010002&displayvalue=true

Setting web-service definition in dispatcher.conf

The web-service config contains the next parameters:

  • [web_service_name] - name to be used from the scheduler UI
  • download_type - can be JSON or CSV
  • username - username to be used for downloading from the provided URL
  • password - password to be used for downloading from the provided URL
  • authURL - to be used instead of username/password, when a value is assigned to this variable, the dispatcher will append the provided value at the end of the Job’s URL, and will not try to authenticate using username/password
  • SonarGateway_Address - IP-address/domain-name of the server running sonar syslog (to be used with JSON). In most cases will be “localhost”
  • SonarGateway_Port - port configured on SonarGateway to handle the specific JSON input

Example:

[MyWebService]
download_type = JSON
username = joesmith
password = password
authURL =
SonarGateway_Address = localhost
SonarGateway_Port = 10532

When importing CSV files the next pattern will be used for the file names when saved to the “incoming” folder:

sonar-web-service-<web_service_name>-<timestamp>-<hash>.csv

Legend:

  • sonar-web-service: This is a fix string used to identify that the source of the file is the web-service
  • web_service_name: This is the web service section name in dispatcher.conf (used for this download)
  • timestamp: in “yyyymmddhhMMss” format (for example: 20170919234915)
  • Hash: ten character hashed uuid (for example: 1617e8d157)

Archiving and Purging Reports

Reports generated by the dispatcher are stored on a local/mounted drive as defined during the system setup.

In order to manage the size usage of reports the “delete_days_interval” parameter in the dispatcher.config is set. Any report that is “older” than the number of days set will be deleted. In order to not delete any report delete_days_interval should be set to “-1”


Archiving reports:

The dispatcher provides an option to archive reports to a SCP destination.

There are 2 config parameters that govern this functionality:

  • archive_copy_dest
  • archive_after_hours

archive_copy_dest refers to a SCP section in the dispatcher.conf where the scp parameters are defined, for example:

[section_name]
copy_host = 192.168.1.1
copy_port = 22
copy_username = user1
copy_password = password1
copy_keyfile =
copy_dest = /home/example

Archive_after_hours sets the number of hours a report needs to be “older than” in order to be archived. In order to disable the archiving set this value to “-1”

The archiving is done on an incremental basis, hence only reports that haven’t been archived previously (i.e. younger than the last archive time) and older than archive_after_hours will be archived.


Cloud backup of Reports

When a cloud storage is defined for DCAP Central an automatic backup of all reports will be executed on a daily basis, using a cron-job that is created during the system setup.

All reports generated by the system are uploaded to the cloud storage.

In order to purge reports from the cloud storage a purge policy needs to be defined on the lmrm__scheduler.lmrm_dispatched_processed collection.

Reports older than the time defined in that policy will be deleted from the cloud storage once a day

Concurrency Model

The Dispatcher has the capability of running jobs concurrently.

This option can be turned on/off using the config variable: multi_process

It accepts 2 options:True/False

When set to True the dispatcher will run jobs concurrently. Once it is turned on, all dispatcher jobs will be executed using this mechanism.

The number and operation of the jobs is managed using the definition in the sonar_resource_pool section. Note: sonar_resource_pool is a reserved name, it should not be changed, nor used for other sections.

The section contains the next parameters:

[sonar_resource_pool]
section_type = recource_pool
pool_size = 4
check_pool_period = 5
max_execution_time = 60

Legend:

  • section_type - specifying the type of the section
  • pool_size - base number of concurrent jobs
  • max_execution_time - number of minutes a job is allowed to run before auto-termination
  • check_pool_period - number of minutes between check cycles, used for auto-incrementing the number of concurrent jobs (see details below)

If the default pool is not defined, or has definition errors, the next values are to be used by the dispatcher:

  • pool_size = 1
  • check_pool_period = 5 - in minutes
  • max_execution_time = 60 - in minutes

Overriding the max execution time:

There may be cases the user would like to limit a specific job to a lower limit than the maximum time set in the configuration. It can be done by specifying the “autoCancel” parameter in the job’s URL.

Note: if the value set for autoCancel is larger than the one in the configuration, it will be overwritten by the dispatcher and the value from the configuration will be used.

for example:

https://localhost:8443/Gateway?name=timeout test&col=full_sql&type=agg&output=csv&**autoCancel=30**&published_by=admin&mongoUri=bW9uZ29kYjovLyR7dXNlcm5hbWV9OiR7cGFzc3dvcmR9QGxvY2FsaG9zdDoyNzExNy9hZG1pbg==&host=localhost&port=27117&db=sonargd&sdb=lmrm__sonarg

https://localhost:8443/Gateway?name=timeout test&col=full_sql&type=agg&output=csv&**autoCancel=30**&published_by=admin&mongoUri=bW9uZ29kYjovLyR7dXNlcm5hbWV9OiR7cGFzc3dvcmR9QGxvY2FsaG9zdDoyNzExNy9hZG1pbg==&host=localhost&port=27117&db=sonargd&sdb=lmrm__sonarg

Automatic incrementation of the allowed number of concurrent jobs

As mentioned earlier there is a “check” cycle for the dispatcher to monitor the number of running jobs. The “check_pool_period” parameter is used for the mechanism that allow the dispatcher to increment the number of jobs on top of the “pool_size” when the pull is fully utilized.

The mechanism works as follows: Every cycle (for example we can use check_pool_period = 5 minutes) the dispatcher reviews the existing jobs, if the number of concurrent jobs running is equal to or larger than the pool_size, the dispatcher checks if any of the running jobs has been started during the past cycle, if none of the jobs was started during the last cycle (i.e. all have been running for more than 5 minutes in this example) the dispatcher is allowed to increment the actual pool-size by one (i.e. if there’s a job in the queue it will be started in another concurrent process). Therefore the actual pool_size is derived from the combination of all 3 config parameters.

Advanced Handling of Reports and CSVs

The dispatcher provides a set of config parameters (in the dispatcher.conf) that enable the system to control the output of report(s) generating jobs.

max_CSV_line_limit - is a limit set to prevent huge PDF files from being generated.

When the CSV file that has been returned as the report result exceeds this limit the output will be a CSV file, even if the job definition is set to PDF .

max_CSV_line_split - is a value that controls the size of CSV files. Since some applications have limits on the number of CSV lines this parameter is used to ensure those limits are not exceeded. When the downloaded CSV report exceeds this value it will be split into multiple CSV files, named <name>_1.csv, <name>_2.csv etc.

The files will be delivered in a zip archive regardless of the bulk_email_report settings (see below)

bulk_email_report - is used when multiple jobs with the same name are scheduled at the same time.

In this case the dispatcher gathers the results of all reports and handles them according to the settings below:

  • separate - each report will be handles as a separate report, regardless of the other jobs/reports
  • zipped - each report will be generated separately and then all will be added to a single zip archive for delivery.
  • merged - this only applies to PDF reports, the dispatcher will combine the outputs from all individual reports into a single PDF report for delivery. Note the max_CSV_line_limit still applies to the combination of all reports.

Mixed reports types for zipped & merged: The dispatcher can handle a mixed type of CSV/PDF reports. In both cases a zip archive will be delivered containing all the reports, in the case of merged, the PDF will be merged to a single file.