8. Additional commands¶

8.1 RSE Core commands¶

8.1.1 Check the cluster health¶

Command:

curl -XGET -H "Content-Type: application/json" http://localhost:9200/_cluster/health?pretty=true

Expected output:

{
   "cluster_name" : "rsecore",
   "status" : "green",
   "timed_out" : false,
   "number_of_nodes" : 3,
   "number_of_data_nodes" : 3,
   "active_primary_shards" : 10,
   "active_shards" : 20,
   "relocating_shards" : 0,
   "initializing_shards" : 0,
   "unassigned_shards" : 0,
   "delayed_unassigned_shards" : 0,
   "number_of_pending_tasks" : 0,
   "number_of_in_flight_fetch" : 0,
   "task_max_waiting_in_queue_millis" : 0,
   "active_shards_percent_as_number" : 100.0
}

8.1.2 Speed up lifecycly policy check¶

Command to set it to 1 minute:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/_cluster/settings --data '
{
   "transient": {
     "indices.lifecycle.poll_interval": "1m"
   }
}'

8.1.3 Set lifecycly policy speed to default¶

Command to reset the policy:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/_cluster/settings --data '
{
   "transient": {
     "indices.lifecycle.poll_interval": null
   }
}'

8.1.4 List indexes¶

Command to list the indexes:

curl -XGET 'localhost:9200/_cat/indices'

8.1.5 Check cluster diskspace¶

Command to list the cluster diskspace:

curl -XGET 'localhost:9200/_cat/allocation?v&pretty'

8.1.6 Filter on host and time¶

Adjust size for more results.

Command to filter on host and time:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must": [ { "match": { "HOST_FROM": "172.16.30.1" } }, { "range": { "R_ISODATE": { "gte": "2022-01-13T22:45:39.493+00:00" } } } ] } } , "size": 3 }' | jq

8.1.7 View top 10 results¶

Command to view top 10 messages:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search?pretty

8.1.8 View the mapping of the fields¶

Command to view mapping of the fields:

curl -X GET http://127.0.0.1:9200/rse*/_mapping?pretty

8.1.9 Search between times¶

Adjust size for more results.

Command to view output between a start and top time:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must": [ { "match": { "HOST_FROM": "172.16.30.1" } }, { "range": { "R_ISODATE": { "gte": "2022-01-13T22:45:39.493+00:00", "lte": "2022-01-17T22:45:39.493+00:00" } } } ] } } , "size": 3 }' | jq

8.1.10 Search uniq MAC adresses from DHCP index¶

Command to view output of uniq MAC adresses from a DHCP index:

Requires logstash to index

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/logstash-rsx-dhcp*/_search?size=10000 -d '{ "query" : { "bool" : { "should": [ { "match": { "Host_Name": "*NUC00*" } }, { "range": { "@timestamp": { "gte": "now-1d/d" } } } ] } } }' | jq | grep MAC_Address | sort | uniq -d

8.1.11 View 2 exact terms¶

Command to view multiple exact terms:

curl -X POST http://127.0.0.1:9200/rse*/_search -H 'Content-Type:application/json' -d '{
"query": {
  "terms" : {
    "HOST_FROM" : [ "172.16.30.1", "172.16.30.24" ]
    }
  }
}' | jq

8.1.12 View 1 exact term¶

Command to view 1 exact term:

curl -X POST http://127.0.0.1:9200/rse*/_search -H 'Content-Type:application/json' -d  '{
"query": {
  "term" : {
   "HOST_FROM" : "172.16.30.1"
   }
  }
}' | jq

8.1.13 Flush indexes¶

Command to start the flush process of an index makes sure that any data that is currently only persisted in the transaction log is also permanently persisted in Lucene.

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/_flush?wait_if_ongoing | jq

or:

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/_flush?wait_if_ongoing | jq

Flush a set or a single index:

Note: use wildcard do group the indexes.

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/rse*/_flush | jq

8.1.14 Delete index¶

Command to delete a single index:

Index = logstash-rsx-2020.03.28

curl -XDELETE http://localhost:9200/logstash-rsx-2020.03.28 | jq

8.1.15 View license¶

Command to view the license:

curl -XGET 'http://localhost:9200/_license?pretty'

8.1.16 Lite search a value on multiple fields¶

Command to filter a single value on all fields:

curl -XGET 'localhost:9200/_all/_search?q=172.16.30.1&pretty'

8.1.17 Lite search a single value for 1 field¶

Command to filter a single value within 1 field:

curl -XGET 'localhost:9200/_all/_search?q=HOST_FROM:172.16.30.1&pretty'

8.1.18 Example searches¶

Create search query for message field:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "match" : { "MESSAGE": "172.16.30.1" } } }' | jq

or

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must": { "match": { "MESSAGE": "172.16.30.1" } } } } }' | jq

Exclude result based on a single word:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must_not": { "match": { "MESSAGE": "172.16.30.1" } } } } }' | jq

8.1.19 Advanced searches¶

Command to exclude a value and filter down a host within a specific time range:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must_not" : [ { "match" : { "PROGRAM" : "dhcpd" } } ], "filter" : [ { "term": { "HOST_FROM" : "172.16.30.1" } }, { "range": { "R_ISODATE": { "gte": "2022-08-06T10:13:00.000+00:00", "lte": "2022-08-06T10:20:00.000+00:00" } } } ] } } , "size": 300 }' | jq

Command to filter down a value within a specific time range using OR:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "should": [ { "match": { "MESSAGE": "172.16.30.1" } }, { "range": { "R_ISODATE": { "gt": "2022-08-06T10:13:00.000+00:00", "lt": "2022-08-06T10:20:00.000+00:00||+1M" } } } ] } } }' | jq

Command to filter down a value within a specific time range using AND (This query uses authentication):

curl -XGET --header 'Content-Type: application/json' http://elastic:elastic@localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must": [ { "match": { "MESSAGE": "marcel" } }, { "range": { "ISODATE": { "gt": "2022-08-12T06:50:14+00:00", "lt": "2022-08-12T06:52:14+00:00" } } } ] } } , "size": 300 }' | jq -r -c '.hits.hits[]._source.MESSAGE'

Command to exclude a value and filter down multiple hosts within a specific time range:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must_not" : [ { "match" : { "PROGRAM" : "dhcpd" } } ], "filter" : [ { "terms": { "HOST_FROM" : [ "172.16.30.1", "172.16.30.24" ] } }, { "range": { "R_ISODATE": { "gte": "2022-08-06T10:13:00.000+00:00", "lte": "2022-08-06T10:20:00.000+00:00" } } } ] } } , "size": 300 }' | jq

Search for value on multiple fields:

Note: Both the fields must match the value.

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "multi_match" : { "query": "172.16.30.1", "fields": [ "MESSAGE", "HOST_FROM" ] } } }' | jq

Search results after data and time with a value using OR:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "should": [ { "match": { "MESSAGE": "172.16.30.1" } }, { "range": { "R_ISODATE": { "gte": "2022-08-06T10:13:00.000+00:00" } } } ] } } }' | jq

Search results after data and time with a value using AND:

curl -XGET --header 'Content-Type: application/json' http://elastic:elastic@localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "must": [ { "match": { "MESSAGE": "marcel" } }, { "match": { "MESSAGE": "VPN" } }, { "range": { "ISODATE": { "gt": "2022-08-12T06:50:14+00:00", "lt": "2022-08-12T06:52:14+00:00" } } } ] } } , "size": 300 }' | jq -r -c '.hits.hits[]._source.MESSAGE'

Search results of the last hour with a value:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "bool" : { "should": [ { "match": { "MESSAGE": "172.16.30.1" } }, { "range": { "R_ISODATE": { "gte": "now-1h" } } } ] } } }' | jq

8.1.20 Validate query’s¶

Check if query’s are valid:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_validate/query -d '{ "query" : { "match" : { "MESSAGE": "172.16.30.1" } } }' | jq

Check if query is valid with explaination:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_validate/query?explain -d '{ "query" : { "match" : { "MESSAGE": "172.16.30.1" } } } }' | jq

8.1.21 Sort results¶

Filter value when using sort:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse*/_search -d '{ "query" : { "match" : { "MESSAGE": "172.16.30.1" } }, "sort": { "_score": { "order": "desc" } } }' | jq

Filter value when using 2 sorts:

curl -XGET --header 'Content-Type: application/json' 'http://localhost:9200/rse*/_search?sort=R_ISODATE:desc&sort=_score&q=172.16.30.1' | jq

8.1.22 Indexes and aliases¶

Create a index:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rse-dummy | jq

Create a alias on a index:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rse-dummy/_alias/rse-dummy2 | jq

View alias:

curl -XGET --header 'Content-Type: application/json' http://localhost:9200/rse-dummy/_alias/* | jq

Example alias usage:

curl -XDELETE --header "Content-Type: application/json" http://localhost:9200/rsx-netflow*

and:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rsx-netflow-000001?pretty -d ' { "aliases": { "rsx-netflow":{ "is_write_index": true } } }'

8.1.23 Refresh indexes¶

Refresh all indexes:

curl -XPOST --header 'Content-Type: application/json' http://localhost:9200/_refresh | jq

Change refresh of index to 30 seconds:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rse-dummy/_settings -d '{ "settings": { "refresh_interval": "30s" }}' | jq

Disable refresh interval for index:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rse-dummy/_settings -d '{ "settings": { "refresh_interval": "-1" }}' | jq

Restore default refresh interval for index:

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/rse-dummy/_settings -d '{ "settings": { "refresh_interval": "1s" }}' | jq

8.1.24 Example lifecycle policy¶

curl -XPUT --header 'Content-Type: application/json' http://localhost:9200/_ilm/policy/netflow-policy -d ' { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "14d" } } }, "delete": { "min_age": "14d", "actions": { "delete": { "delete_searchable_snapshot": true } } } } } }' | jq