Common-processors¶

Find below the list.

AddFields¶

Add one or more field to records

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.AddFields

Tags¶

record, fields, Add

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
conflict.resolution.policy	What to do when a field with the same name already exists ?	overwrite_existing (if field already exist), keep_only_old_field (keep only old field)	keep_only_old_field	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
Name of the field to add	Value of the field to add	Add a field to the record with the specified value. Expression language can be used.You can not add a field that end with ‘.type’ as this suffix is used to specify the type of fields to add		null	true
Name of the field to add with the suffix ‘.field.type’	Type of the field to add	Add a field to the record with the specified type. These properties are only used if a correspondant property without the suffix ‘.field.type’ is already defined. If this property is not defined, default type for adding fields is String.You can only use Logisland predefined type fields.	NULL, STRING, INT, LONG, ARRAY, FLOAT, DOUBLE, BYTES, RECORD, MAP, ENUM, BOOLEAN, UNION, DATETIME	STRING	false
Name of the field to add with the suffix ‘.field.name’	Name of the field to add using expression language	Add a field to the record with the specified name (which is evaluated using expression language). These properties are only used if a correspondant property without the suffix ‘.field.name’ is already defined. If this property is not defined, the name of the field to add is the key of the first dynamic property (which is the main and only required dynamic property).		null	true

Extra informations¶

Add one or more field with constant value or dynamic value using the expression-language.Some examples of settings:

newStringField: bonjour
newIntField: 14
newIntField.field.type: INT

Would add those fields in record :

Field{name='newStringField', type='STRING', value='bonjour'}
Field{name='newIntField', type='INT', value=14}

Here a second example using expression language, once for the value, once for the key. Note that you can use for both.We suppose that our record got already those fields :

Field{name='field1', type='STRING', value='bonjour'}
Field{name='field2', type='INT', value=14}

This settings : .. code:

newStringField: ${field1 + "-" + field2}
fieldToCalulateKey: 555
fieldToCalulateKey.field.name: ${"_" + field1 + "-"}

Would add those fields in record :

Field{name='newStringField', type='STRING', value='bonjour-14'}
Field{name='_bonjour-', type='STRING', value='555'}

As you probably notice, you can not add fields with name ending by either ‘.field.name’ either ‘.field.type’ because they are suffix are used to sort dynamic properties. But if you really want to do this a workaround is to specify the name of the field oui expression language, for example this settings would work:

fieldWithReservedSuffix: bonjour
fieldWithReservedSuffix.field.type: INT
fieldWithReservedSuffix.field.type: myfield.endind.with.reserved.suffix.field.type

ApplyRegexp¶

This processor is used to create a new set of fields from one field (using regexp).

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ApplyRegexp

Tags¶

parser, regex, log, record

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
conflict.resolution.policy	What to do when a field with the same name already exists ?	overwrite_existing (if field already exist), keep_only_old_field (keep only old field)	keep_only_old_field	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
alternative regex & mapping	another regex that could match	This processor is used to create a new set of fields from one field (using regexp).		null	true

Extra informations¶

This processor is used to create a new set of fields from one field (using regexp).

BulkPut¶

Indexes the content of a Record in a Datastore using bulk processor

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.datastore.BulkPut

Tags¶

datastore, record, put, bulk

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the Expression Language .

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
datastore.client.service	The instance of the Controller Service to use for accessing datastore.		null	false	false
default.collection	The name of the collection/index/table to insert into		null	false	true
timebased.collection	do we add a date suffix	no (no date added to default index), today (today’s date added to default index), yesterday (yesterday’s date added to default index)	no	false	false
date.format	simple date format for date suffix. default : yyyy.MM.dd		yyyy.MM.dd	false	false
collection.field	the name of the event field containing es index name => will override index value if set		null	false	true

Extra informations¶

Indexes the content of a Record in a Datastore using bulk processor.

CheckAlerts¶

Add one or more records representing alerts. Using a datastore.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.alerting.CheckAlerts

Tags¶

record, alerting, thresholds, opc, tag

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
max.cpu.time	maximum CPU time in milliseconds allowed for script execution.	100	false	false
max.memory	maximum memory in Bytes which JS executor thread can allocate	51200	false	false
allow.no.brace	Force, to check if all blocks are enclosed with curly braces “”{}”“. Explanation: all loops (for, do-while, while, and if-else, and functions should use braces, because poison_pill() function will be inserted after each open brace ""{"", to ensure interruption checking. Otherwise simple code like: while(true) while(true) { // do nothing } or even: while(true) cause unbreakable loop, which force this sandbox to use {@link Thread#stop()} which make JVM unstable. Properly writen code (even in bad intention) like: while(true) { while(true) { // do nothing }} will be changed into: while(true) {poison_pill(); while(true) {poison_pill(); // do nothing } } which finish nicely when interrupted. For legacy code, this check can be turned off, but with no guarantee, the JS thread will gracefully finish when interrupted.	false	false	false
max.prepared.statements	The size of prepared statements LRU cache. If 0, this is disabled. Each statements when setMaxCPUTime(long) is set is prepared to quit itself when time exceeded. To execute only once this procedure per statement set this value. When setMaxCPUTime(long) is set 0, this value is ignored.	30	false	false
datastore.client.service	The instance of the Controller Service to use for accessing datastore.	null	false	false
datastore.cache.collection	The collection where to find cached objects	test	false	false
js.cache.service	The cache service to be used to store already sanitized JS expressions. If not specified a in-memory unlimited hash map will be used.	null	false	false
output.record.type	the type of the output record	event	false	false
profile.activation.condition	A javascript expression that activates this alerting profile when true	0==0	false	false
alert.criticity	from 0 to …	0	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
field to add	a default value	Add a field to the record with the default value		null	false

Extra informations¶

Add one or more records representing alerts. Using a datastore.

CheckThresholds¶

Compute threshold cross from given formulas.

each dynamic property will return a new record according to the formula definition
the record name will be set to the property name
the record time will be set to the current timestamp

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.alerting.CheckThresholds

Tags¶

record, threshold, tag, alerting

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
max.cpu.time	maximum CPU time in milliseconds allowed for script execution.	100	false	false
max.memory	maximum memory in Bytes which JS executor thread can allocate	51200	false	false
allow.no.brace	Force, to check if all blocks are enclosed with curly braces “”{}”“. Explanation: all loops (for, do-while, while, and if-else, and functions should use braces, because poison_pill() function will be inserted after each open brace ""{"", to ensure interruption checking. Otherwise simple code like: while(true) while(true) { // do nothing } or even: while(true) cause unbreakable loop, which force this sandbox to use {@link Thread#stop()} which make JVM unstable. Properly writen code (even in bad intention) like: while(true) { while(true) { // do nothing }} will be changed into: while(true) {poison_pill(); while(true) {poison_pill(); // do nothing } } which finish nicely when interrupted. For legacy code, this check can be turned off, but with no guarantee, the JS thread will gracefully finish when interrupted.	false	false	false
max.prepared.statements	The size of prepared statements LRU cache. If 0, this is disabled. Each statements when setMaxCPUTime(long) is set is prepared to quit itself when time exceeded. To execute only once this procedure per statement set this value. When setMaxCPUTime(long) is set 0, this value is ignored.	30	false	false
datastore.client.service	The instance of the Controller Service to use for accessing datastore.	null	false	false
datastore.cache.collection	The collection where to find cached objects	test	false	false
js.cache.service	The cache service to be used to store already sanitized JS expressions. If not specified a in-memory unlimited hash map will be used.	null	false	false
output.record.type	the type of the output record	event	false	false
record.ttl	How long (in ms) do the record will remain in cache	30000	false	false
min.update.time.ms	The minimum amount of time (in ms) that we expect between two consecutive update of the same threshold record	200	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
field to add	a default value	Add a field to the record with the default value		null	false

Extra informations¶

Compute threshold cross from given formulas.

each dynamic property will return a new record according to the formula definition
the record name will be set to the property name
the record time will be set to the current timestamp

ComputeTags¶

Compute tag cross from given formulas.

each dynamic property will return a new record according to the formula definition
the record name will be set to the property name
the record time will be set to the current timestamp

a threshold_cross has the following properties : count, sum, avg, time, duration, value

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.alerting.ComputeTags

Tags¶

record, fields, Add

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
max.cpu.time	maximum CPU time in milliseconds allowed for script execution.	100	false	false
max.memory	maximum memory in Bytes which JS executor thread can allocate	51200	false	false
allow.no.brace	Force, to check if all blocks are enclosed with curly braces “”{}”“. Explanation: all loops (for, do-while, while, and if-else, and functions should use braces, because poison_pill() function will be inserted after each open brace ""{"", to ensure interruption checking. Otherwise simple code like: while(true) while(true) { // do nothing } or even: while(true) cause unbreakable loop, which force this sandbox to use {@link Thread#stop()} which make JVM unstable. Properly writen code (even in bad intention) like: while(true) { while(true) { // do nothing }} will be changed into: while(true) {poison_pill(); while(true) {poison_pill(); // do nothing } } which finish nicely when interrupted. For legacy code, this check can be turned off, but with no guarantee, the JS thread will gracefully finish when interrupted.	false	false	false
max.prepared.statements	The size of prepared statements LRU cache. If 0, this is disabled. Each statements when setMaxCPUTime(long) is set is prepared to quit itself when time exceeded. To execute only once this procedure per statement set this value. When setMaxCPUTime(long) is set 0, this value is ignored.	30	false	false
datastore.client.service	The instance of the Controller Service to use for accessing datastore.	null	false	false
datastore.cache.collection	The collection where to find cached objects	test	false	false
js.cache.service	The cache service to be used to store already sanitized JS expressions. If not specified a in-memory unlimited hash map will be used.	null	false	false
output.record.type	the type of the output record	event	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
field to add	a default value	Add a field to the record with the default value		null	false

Extra informations¶

Compute tag cross from given formulas.

each dynamic property will return a new record according to the formula definition
the record name will be set to the property name
the record time will be set to the current timestamp

a threshold_cross has the following properties : count, sum, avg, time, duration, value

ConvertFieldsType¶

Converts a field value into the given type. does nothing if conversion is not possible

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ConvertFieldsType

Tags¶

type, fields, update, convert

Properties¶

This component has no required or optional properties.

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
field	the new type	convert field value into new type		null	true

Extra informations¶

Converts a field value into the given type. does nothing if conversion is not possible.

ConvertSimpleDateFormatFields¶

Convert one or more field representing a date into a Unix Epoch Time (time in milliseconds since &st January 1970, 00:00:00 GMT)…

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ConvertSimpleDateFormatFields

Tags¶

record, fields, Add

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
conflict.resolution.policy	What to do when a field with the same name already exists ?	overwrite_existing (if field already exist), keep_only_old_field (keep only old field)	keep_only_old_field	false	false
input.date.format	Simple date format representation of the input field to convert		null	false	false
timezone	Specify the timezone (default is CET)		CET	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
field name to add	value to convert into Epoch timestamp using given input.date.format	Add a field to the record with the name, converting value using java SimpleDateFormat		null	true

Extra informations¶

Convert one or more field representing a date into a Unix Epoch Time (time in milliseconds since &st January 1970, 00:00:00 GMT)…

DebugStream¶

This is a processor that logs incoming records

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.DebugStream

Tags¶

record, debug

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
event.serializer	the way to serialize event	json (serialize events as json blocs), string (serialize events as toString() blocs)	json	false	false

Extra informations¶

This is a processor that logs incoming records.

EnrichRecords¶

Enrich input records with content indexed in datastore using multiget queries. Each incoming record must be possibly enriched with information stored in datastore. The plugin properties are :

es.index (String) : Name of the datastore index on which the multiget query will be performed. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.
record.key (String) : Name of the field in the input record containing the id to lookup document in elastic search. This field is mandatory.
es.key (String) : Name of the datastore key on which the multiget query will be performed. This field is mandatory.
includes (ArrayList<String>) : List of patterns to filter in (include) fields to retrieve. Supports wildcards. This field is not mandatory.
excludes (ArrayList<String>) : List of patterns to filter out (exclude) fields to retrieve. Supports wildcards. This field is not mandatory.

Each outcoming record holds at least the input record plus potentially one or more fields coming from of one datastore document.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.datastore.EnrichRecords

Tags¶

datastore, enricher

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the Expression Language .

allowable-values¶
Name	Description	Default Value	Sensitive	EL
datastore.client.service	The instance of the Controller Service to use for accessing datastore.	null	false	false
record.key	The name of field in the input record containing the document id to use in ES multiget query	null	false	true
includes.field	The name of the ES fields to include in the record.		false	true
excludes.field	The name of the ES fields to exclude.	N/A	false	false
type.name	The typle of record to look for	null	false	true
collection.name	The name of the collection to look for	null	false	true

Extra informations¶

Enrich input records with content indexed in datastore using multiget queries. Each incoming record must be possibly enriched with information stored in datastore. The plugin properties are :

es.index (String) : Name of the datastore index on which the multiget query will be performed. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.
record.key (String) : Name of the field in the input record containing the id to lookup document in elastic search. This field is mandatory.
es.key (String) : Name of the datastore key on which the multiget query will be performed. This field is mandatory.
includes (ArrayList<String>) : List of patterns to filter in (include) fields to retrieve. Supports wildcards. This field is not mandatory.
excludes (ArrayList<String>) : List of patterns to filter out (exclude) fields to retrieve. Supports wildcards. This field is not mandatory.

Each outcoming record holds at least the input record plus potentially one or more fields coming from of one datastore document.

EvaluateJsonPath¶

Evaluates one or more JsonPath expressions against the content of a FlowFile. The results of those expressions are assigned to Records Fields depending on configuration of the Processor. JsonPaths are entered by adding user-defined properties; the name of the property maps to the Field Name into which the result will be placed. The value of the property must be a valid JsonPath expression. A Return Type of ‘auto-detect’ will make a determination based off the configured destination. If the JsonPath evaluates to a JSON array or JSON object and the Return Type is set to ‘scalar’ the Record will be routed to error. A Return Type of JSON can return scalar values if the provided JsonPath evaluates to the specified value. If the expression matches nothing, Fields will be created with empty strings as the value

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.EvaluateJsonPath

Tags¶

JSON, evaluate, JsonPath

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
return.type	Indicates the desired return type of the JSON Path expressions. Selecting ‘auto-detect’ will set the return type to ‘json’ or ‘scalar’	json, scalar	scalar	false	false
path.not.found.behavior	Indicates how to handle missing JSON path expressions. Selecting ‘warn’ will generate a warning when a JSON path expression is not found.	warn, ignore	ignore	false	false
Null Value Representation	Indicates the desired representation of JSON Path expressions resulting in a null value.	empty string, the string ‘null’	empty string	false	false
json.input.field.name	the name of the field containing the json string		record_value	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
A Record field	A JsonPath expression	will be set to any JSON objects that match the JsonPath.		null	false

Extra informations¶

Evaluates one or more JsonPath expressions against the content of a FlowFile. The results of those expressions are assigned to Records Fields depending on configuration of the Processor. JsonPaths are entered by adding user-defined properties; the name of the property maps to the Field Name into which the result will be placed. The value of the property must be a valid JsonPath expression. A Return Type of ‘auto-detect’ will make a determination based off the configured destination. If the JsonPath evaluates to a JSON array or JSON object and the Return Type is set to ‘scalar’ the Record will be routed to error. A Return Type of JSON can return scalar values if the provided JsonPath evaluates to the specified value. If the expression matches nothing, Fields will be created with empty strings as the value.

ExpandMapFields¶

Expands the content of a MAP field to the root.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ExpandMapFields

Tags¶

record, fields, Expand, Map

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
fields.to.expand	Comma separated list of fields of type map that will be expanded to the root		null	false	false
conflict.resolution.policy	What to do when a field with the same name already exists ?	overwrite_existing (if field already exist), keep_only_old_field (keep only old field)	keep_only_old_field	false	false

Extra informations¶

Expands the content of a MAP field to the root.

FilterRecords¶

Keep only records based on a given field value

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.FilterRecords

Tags¶

record, fields, remove, delete

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
field.name	the field name		record_id	false	false
field.value	the field value to keep		null	false	false

Extra informations¶

Keep only records based on a given field value.

FlatMap¶

Converts each field records into a single flatten record…

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.FlatMap

Tags¶

record, fields, flatmap, flatten

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
keep.root.record	do we add the original record in	true	false	false
copy.root.record.fields	do we copy the original record fields into the flattened records	true	false	false
leaf.record.type	the new type for the flattened records if present		false	false
concat.fields	comma separated list of fields to apply concatenation ex : $rootField/$leaffield	null	false	false
concat.separator	returns $rootField/$leaf/field	/	false	false
include.position	do we add the original record position in	true	false	false

Extra informations¶

Converts each field records into a single flatten record…

GenerateRandomRecord¶

This is a processor that make random records given an Avro schema

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.GenerateRandomRecord

Tags¶

record, avro, generator

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
avro.output.schema	the avro schema definition for the output serialization	null	false	false
min.events.count	the minimum number of generated events each run	10	false	false
max.events.count	the maximum number of generated events each run	200	false	false

Extra informations¶

This is a processor that make random records given an Avro schema.

ModifyId¶

modify id of records or generate it following defined rules

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ModifyId

Tags¶

record, id, idempotent, generate, modify

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
id.generation.strategy	the strategy to generate new Id	randomUuid (generate a randomUid using java library), hashFields (generate a hash from fields), fromFields (generate a string from java pattern and fields), typetimehash (generate a concatenation of type, time and a hash from fields (as for generate_hash strategy))	randomUuid	false	false
fields.to.hash	the comma separated list of field names (e.g. : ‘policyid,date_raw’		record_value	false	false
hash.charset	the charset to use to hash id string (e.g. ‘UTF-8’)		UTF-8	false	false
hash.algorithm	the algorithme to use to hash id string (e.g. ‘SHA-256’	SHA-384, SHA-224, SHA-256, MD2, SHA, SHA-512, MD5	SHA-256	false	false
java.formatter.string	the format to use to build id string (e.g. ‘%4$2s %3$2s %2$2s %1$2s’ (see java Formatter)		null	false	false
language.tag	the language to use to format numbers in string	aa, ab, ae, af, ak, am, an, ar, as, av, ay, az, ba, be, bg, bh, bi, bm, bn, bo, br, bs, ca, ce, ch, co, cr, cs, cu, cv, cy, da, de, dv, dz, ee, el, en, eo, es, et, eu, fa, ff, fi, fj, fo, fr, fy, ga, gd, gl, gn, gu, gv, ha, he, hi, ho, hr, ht, hu, hy, hz, ia, id, ie, ig, ii, ik, in, io, is, it, iu, iw, ja, ji, jv, ka, kg, ki, kj, kk, kl, km, kn, ko, kr, ks, ku, kv, kw, ky, la, lb, lg, li, ln, lo, lt, lu, lv, mg, mh, mi, mk, ml, mn, mo, mr, ms, mt, my, na, nb, nd, ne, ng, nl, nn, no, nr, nv, ny, oc, oj, om, or, os, pa, pi, pl, ps, pt, qu, rm, rn, ro, ru, rw, sa, sc, sd, se, sg, si, sk, sl, sm, sn, so, sq, sr, ss, st, su, sv, sw, ta, te, tg, th, ti, tk, tl, tn, to, tr, ts, tt, tw, ty, ug, uk, ur, uz, ve, vi, vo, wa, wo, xh, yi, yo, za, zh, zu	en	false	false

Extra informations¶

modify id of records or generate it following defined rules.

MultiGet¶

Retrieves a content from datastore using datastore multiget queries. Each incoming record contains information regarding the datastore multiget query that will be performed. This information is stored in record fields whose names are configured in the plugin properties (see below) :

collection (String) : name of the datastore collection on which the multiget query will be performed. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.

type (String) : name of the datastore type on which the multiget query will be performed. This field is not mandatory.

ids (String) : comma separated list of document ids to fetch. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.

includes (String) : comma separated list of patterns to filter in (include) fields to retrieve. Supports wildcards. This field is not mandatory.

excludes (String) : comma separated list of patterns to filter out (exclude) fields to retrieve. Supports wildcards. This field is not mandatory.

Each outcoming record holds data of one datastore retrieved document. This data is stored in these fields :

collection (same field name as the incoming record) : name of the datastore collection.

type (same field name as the incoming record) : name of the datastore type.

id (same field name as the incoming record) : retrieved document id.

a list of String fields containing :

field name : the retrieved field name

field value : the retrieved field value

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.datastore.MultiGet

Tags¶

datastore, get, multiget

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
datastore.client.service	The instance of the Controller Service to use for accessing datastore.	null	false	false
collection.field	the name of the incoming records field containing es collection name to use in multiget query.	null	false	false
type.field	the name of the incoming records field containing es type name to use in multiget query	null	false	false
ids.field	the name of the incoming records field containing es document Ids to use in multiget query	null	false	false
includes.field	the name of the incoming records field containing es includes to use in multiget query	null	false	false
excludes.field	the name of the incoming records field containing es excludes to use in multiget query	null	false	false

Extra informations¶

Retrieves a content from datastore using datastore multiget queries. Each incoming record contains information regarding the datastore multiget query that will be performed. This information is stored in record fields whose names are configured in the plugin properties (see below) :

collection (String) : name of the datastore collection on which the multiget query will be performed. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.

type (String) : name of the datastore type on which the multiget query will be performed. This field is not mandatory.

ids (String) : comma separated list of document ids to fetch. This field is mandatory and should not be empty, otherwise an error output record is sent for this specific incoming record.

includes (String) : comma separated list of patterns to filter in (include) fields to retrieve. Supports wildcards. This field is not mandatory.

excludes (String) : comma separated list of patterns to filter out (exclude) fields to retrieve. Supports wildcards. This field is not mandatory.

Each outcoming record holds data of one datastore retrieved document. This data is stored in these fields :

collection (same field name as the incoming record) : name of the datastore collection.

type (same field name as the incoming record) : name of the datastore type.

id (same field name as the incoming record) : retrieved document id.

a list of String fields containing :

field name : the retrieved field name

field value : the retrieved field value

NormalizeFields¶

Changes the name of a field according to a provided name mapping…

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.NormalizeFields

Tags¶

record, fields, normalizer

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
conflict.resolution.policy	what to do when a field with the same name already exists ?	do_nothing (leave record as it was), overwrite_existing (if field already exist), keep_only_old_field (keep only old field and delete the other), keep_both_fields (creates an alias for the new field)	do_nothing	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
alternative mapping	a comma separated list of possible field name	when a field has a name contained in the list it will be renamed with this property field name		null	true

Extra informations¶

Changes the name of a field according to a provided name mapping…

ParseProperties¶

Parse a field made of key=value fields separated by spaces a string like “a=1 b=2 c=3” will add a,b & c fields, respectively with values 1,2 & 3 to the current Record

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.ParseProperties

Tags¶

record, properties, parser

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
properties.field	the field containing the properties to split and treat		null	false	false

Extra informations¶

Parse a field made of key=value fields separated by spaces a string like “a=1 b=2 c=3” will add a,b & c fields, respectively with values 1,2 & 3 to the current Record

RemoveFields¶

Removes a list of fields defined by a comma separated list of field names or keeps only fields defined by a comma separated list of field names.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.RemoveFields

Tags¶

record, fields, remove, delete, keep

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
fields.to.remove	A comma separated list of field names to remove (e.g. ‘policyid,date_raw’). Usage of this property is mutually exclusive with the fields.to.keep property. In any case the technical logisland fields record_id, record_time and record_type are not removed even if specified in the list to remove.		null	false	false
fields.to.keep	A comma separated list of field names to keep (e.g. ‘policyid,date_raw’. All other fields will be removed. Usage of this property is mutually exclusive with the PropertyDescriptor[fields.to.remove] property. In any case the technical logisland fields record_id, record_time and record_type are not removed even if not specified in the list to keep.		null	false	false

Extra informations¶

Removes a list of fields defined by a comma separated list of field names or keeps only fields defined by a comma separated list of field names.

SelectDistinctRecords¶

Keep only distinct records based on a given field

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SelectDistinctRecords

Tags¶

record, fields, remove, delete

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
field.name	the field to distinct records		record_id	false	false

Extra informations¶

Keep only distinct records based on a given field.

SendMail¶

The SendMail processor is aimed at sending an email (like for instance an alert email) from an incoming record. There are three ways an incoming record can generate an email according to the special fields it must embed. Here is a list of the record fields that generate a mail and how they work:

mail_text: this is the simplest way for generating a mail. If present, this field means to use its content (value) as the payload of the mail to send. The mail is sent in text format if there is only this special field in the record. Otherwise, used with either mail_html or mail_use_template, the content of mail_text is the aletrnative text to the HTML mail that is generated.
mail_html: this field specifies that the mail should be sent as HTML and the value of the field is mail payload. If mail_text is also present, its value is used as the alternative text for the mail. mail_html cannot be used with mail_use_template: only one of those two fields should be present in the record.
mail_use_template: If present, this field specifies that the mail should be sent as HTML and the HTML content is to be generated from the template in the processor configuration key html.template. The template can contain parameters which must also be present in the record as fields. See documentation of html.template for further explanations. mail_use_template cannot be used with mail_html: only one of those two fields should be present in the record.

If allow_overwrite configuration key is true, any mail.* (dot format) configuration key may be overwritten with a matching field in the record of the form mail_* (underscore format). For instance if allow_overwrite is true and mail.to is set to config_address@domain.com, a record generating a mail with a mail_to field set to record_address@domain.com will send a mail to record_address@domain.com.

Apart from error records (when he is unable to process the incoming record or to send the mail), this processor is not expected to produce any output records.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SendMail

Tags¶

smtp, email, e-mail, mail, mailer, sendmail, message, alert, html

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
debug	Enable debug. If enabled, debug information are written to stdout.	false	false	false
smtp.server	FQDN, hostname or IP address of the SMTP server to use.	null	false	false
smtp.port	TCP port number of the SMTP server to use.	25	false	false
smtp.security.username	SMTP username.	null	false	false
smtp.security.password	SMTP password.	null	false	false
smtp.security.ssl	Use SSL under SMTP or not (SMTPS). Default is false.	false	false	false
mail.from.address	Valid mail sender email address.	null	false	false
mail.from.name	Mail sender name.	null	false	false
mail.bounce.address	Valid bounce email address (where error mail is sent if the mail is refused by the recipient server).	null	false	false
mail.replyto.address	Reply to email address.	null	false	false
mail.subject	Mail subject.	[LOGISLAND] Automatic email	false	false
mail.to	Comma separated list of email recipients. If not set, the record must have a mail_to field and allow_overwrite configuration key should be true.	null	false	false
allow_overwrite	If true, allows to overwrite processor configuration with special record fields (mail_to, mail_from_address, mail_from_name, mail_bounce_address, mail_replyto_address, mail_subject). If false, special record fields are ignored and only processor configuration keys are used.	true	false	false
html.template	HTML template to use. It is used when the incoming record contains a mail_use_template field. The template may contain some parameters. The parameter format in the template is of the form ${xxx}. For instance ${param_user} in the template means that a field named param_user must be present in the record and its value will replace the ${param_user} string in the HTML template when the mail will be sent. If some parameters are declared in the template, everyone of them must be present in the record as fields, otherwise the record will generate an error record. If an incoming record contains a mail_use_template field, a template must be present in the configuration and the HTML mail format will be used. If the record also contains a mail_text field, its content will be used as an alternative text message to be used in the mail reader program of the recipient if it does not supports HTML.	null	false	false

Extra informations¶

The SendMail processor is aimed at sending an email (like for instance an alert email) from an incoming record. There are three ways an incoming record can generate an email according to the special fields it must embed. Here is a list of the record fields that generate a mail and how they work:

mail_text: this is the simplest way for generating a mail. If present, this field means to use its content (value) as the payload of the mail to send. The mail is sent in text format if there is only this special field in the record. Otherwise, used with either mail_html or mail_use_template, the content of mail_text is the aletrnative text to the HTML mail that is generated.
mail_html: this field specifies that the mail should be sent as HTML and the value of the field is mail payload. If mail_text is also present, its value is used as the alternative text for the mail. mail_html cannot be used with mail_use_template: only one of those two fields should be present in the record.
mail_use_template: If present, this field specifies that the mail should be sent as HTML and the HTML content is to be generated from the template in the processor configuration key html.template. The template can contain parameters which must also be present in the record as fields. See documentation of html.template for further explanations. mail_use_template cannot be used with mail_html: only one of those two fields should be present in the record.

If allow_overwrite configuration key is true, any mail.* (dot format) configuration key may be overwritten with a matching field in the record of the form mail_* (underscore format). For instance if allow_overwrite is true and mail.to is set to config_address@domain.com, a record generating a mail with a mail_to field set to record_address@domain.com will send a mail to record_address@domain.com.

Apart from error records (when he is unable to process the incoming record or to send the mail), this processor is not expected to produce any output records.

SetJsonAsFields¶

The SetJsonAsFields processor reads the content of a string field containing a json string and sets each json attribute as a field of the current record. Note that this could be achieved with the EvaluateJsonPath processor, but this implies to declare each json first level attribute in the configuration and also to know by advance every one of them. Whereas for this simple case, the SetJsonAsFields processor does not require such a configuration and will work with any incoming json, regardless of the list of first level attributes.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SetJsonAsFields

Tags¶

json

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
debug	Enable debug. If enabled, debug information are written to stdout.	false	false	false
json.field	Field name of the string field that contains the json document to parse.	record_value	false	false
keep.json.field	Keep the original json field or not. Default is false so default is to remove the json field.	false	false	false
overwrite.existing.field	Overwrite an existing record field or not. Default is true so default is to remove the conflicting field.	true	false	false
omit.null.attributes	Omit json attributes with null values. Default is false so to set them as null record fields	false	false	false
omit.empty.string.attributes	Omit json attributes with empty string values. Default is false so to set them as empty string record fields	false	false	false

Extra informations¶

The SetJsonAsFields processor reads the content of a string field containing a json string and sets each json attribute as a field of the current record. Note that this could be achieved with the EvaluateJsonPath processor, but this implies to declare each json first level attribute in the configuration and also to know by advance every one of them. Whereas for this simple case, the SetJsonAsFields processor does not require such a configuration and will work with any incoming json, regardless of the list of first level attributes.

SplitField¶

This processor is used to create a new set of fields from one field (using split).

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SplitField

Tags¶

parser, split, log, record

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Allowable Values	Default Value	Sensitive	EL
conflict.resolution.policy	What to do when a field with the same name already exists ?	overwrite_existing (if field already exist), keep_only_old_field (keep only old field)	keep_only_old_field	false	false
split.limit	Specify the maximum number of split to allow		10	false	false
split.counter.enable	Enable the counter of items returned by the split		false	false	false
split.counter.suffix	Enable the counter of items returned by the split		Counter	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
alternative split field	another split that could match	This processor is used to create a new set of fields from one field (using split).		null	true

Extra informations¶

This processor is used to create a new set of fields from one field (using split).

SplitText¶

This is a processor that is used to split a String into fields according to a given Record mapping

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SplitText

Tags¶

parser, regex, log, record

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
value.regex	the regex to match for the message value	null	false	false
value.fields	a comma separated list of fields corresponding to matching groups for the message value	null	false	false
key.regex	the regex to match for the message key	.*	false	false
key.fields	a comma separated list of fields corresponding to matching groups for the message key	record_key	false	false
record.type	default type of record	record	false	false
keep.raw.content	do we add the initial raw content ?	true	false	false
timezone.record.time	what is the time zone of the string formatted date for ‘record_time’ field.	UTC	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
alternative regex & mapping	another regex that could match	this regex will be tried if the main one has not matched. It must be in the form alt.value.regex.1 and alt.value.fields.1		null	true

Extra informations¶

This is a processor that is used to split a String into fields according to a given Record mapping.

SplitTextMultiline¶

No description provided.

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SplitTextMultiline

Tags¶

None.

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
regex	the regex to match	null	false	false
fields	a comma separated list of fields corresponding to matching groups	null	false	false
event.type	the type of event	null	false	false

Extra informations¶

No description provided.

SplitTextWithProperties¶

This is a processor that is used to split a String into fields according to a given Record mapping

Module¶

com.hurence.logisland:logisland-processor-common:1.1.1

Class¶

com.hurence.logisland.processor.SplitTextWithProperties

Tags¶

parser, regex, log, record

Properties¶

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

allowable-values¶
Name	Description	Default Value	Sensitive	EL
value.regex	the regex to match for the message value	null	false	false
value.fields	a comma separated list of fields corresponding to matching groups for the message value	null	false	false
key.regex	the regex to match for the message key	.*	false	false
key.fields	a comma separated list of fields corresponding to matching groups for the message key	record_key	false	false
record.type	default type of record	record	false	false
keep.raw.content	do we add the initial raw content ?	true	false	false
properties.field	the field containing the properties to split and treat	properties	false	false

Dynamic Properties¶

Dynamic Properties allow the user to specify both the name and value of a property.

dynamic-properties¶
Name	Value	Description	Allowable Values	Default Value	EL
alternative regex & mapping	another regex that could match	this regex will be tried if the main one has not matched. It must be in the form alt.value.regex.1 and alt.value.fields.1		null	true

Extra informations¶

This is a processor that is used to split a String into fields according to a given Record mapping.

Common-processors¶

AddFields¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

ApplyRegexp¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

See Also:¶

BulkPut¶

Module¶

Class¶

Tags¶

Properties¶

Extra informations¶

CheckAlerts¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

CheckThresholds¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

ComputeTags¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

ConvertFieldsType¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

ConvertSimpleDateFormatFields¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

DebugStream¶

Module¶

Class¶

Tags¶

Properties¶

Extra informations¶

EnrichRecords¶

Module¶

Class¶

Tags¶

Properties¶

Extra informations¶

EvaluateJsonPath¶

Module¶

Class¶

Tags¶

Properties¶

Dynamic Properties¶

Extra informations¶

ExpandMapFields¶

Module¶

Class¶

Tags¶