12. FAQ¶

12.1 My RSX/RSE installation does not recieve any logging¶

You probably should check the date. If the date is not correct run in the CLI as root:

dpkg-reconfigure tzdata

This allows you to configure the timezone.

The next thing to check is within the Kibana console: Management => Advanced Settings => Timezone for date formatting => setup the right timezone.

12.2 Disk full by Geo2¶

Message in logging:

Jan 27 10:24:50 plisk002.prd.corp syslog-ng[1793]: geoip2(): getaddrinfo failed; gai_error='Name or service not known', ip='', location='/etc/syslog-ng/conf.d/99X-Checkpoint.conf:32:25'
Jan 27 10:24:50 plisk002.prd.corp syslog-ng[1793]: geoip2(): maxminddb error; error='Unknown error code', ip='', location='/etc/syslog-ng/conf.d/99X-Checkpoint.conf:32:25'

Components needed for fix:

File: /etc/syslog-ng/syslog-ng.conf

File destinations:

- d_syslog
- d_error

Log rules:

- log { source(s_src); filter(f_syslog3); destination(d_syslog); };
- log { source(s_src); filter(f_error); destination(d_error); };

Fix:

vi /etc/syslog-ng/syslog-ng.conf

Add rules:

filter geoip_messages_1 { not match("Name or service not known"); };
filter geoip_messages_2 { not match("Unknown error code"); };

Change rules:

-log { source(s_src); filter(f_syslog3); destination(d_syslog); };
-log { source(s_src); filter(f_error); destination(d_error); };
+log { source(s_src); filter(f_syslog3); filter(geoip_messages_1); filter(geoip_messages_2); destination(d_syslog); };
+log { source(s_src); filter(f_error); filter(geoip_messages_1); filter(geoip_messages_2); destination(d_error); };

12.3 Kibana not loaded after upgrade¶

Restarting the server will solve this problem. Some report that a restart of the Kibana or Elasticsearch will fix the issue.

service elasticsearch restart
service kibana restart

12.4 Data too large, data for [<http_request>] (JVM heap size)¶

Error message:

tom@plisx002:~$ curl -X GET 'http://localhost:9200/_cat/health?v'
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1014538592/967.5mb], which is larger than the limit of [986061209/940.3mb], real usage: [1014538592/967.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=3057213/2.9mb, in_flight_requests=0/0b, accounting=261018719/248.9mb]","bytes_wanted":1014538592,"bytes_limit":986061209,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1014538592/967.5mb], which is larger than the limit of [986061209/940.3mb], real usage: [1014538592/967.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=3057213/2.9mb, in_flight_requests=0/0b, accounting=261018719/248.9mb]","bytes_wanted":1014538592,"bytes_limit":986061209,"durability":"PERMANENT"},"status":429}

Increase memory fix:

nano /etc/elasticsearch/jvm.options

Edit:

--Xms1g
--Xmx1g
+-Xms6g
+-Xmx6g

12.5 Syslog-NG 3.27.1 breaks with new upgrade on Ubuntu 18.04 and 20.04¶

Error message:

dpkg: error processing package syslog-ng-mod-sql (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of syslog-ng-mod-redis:
 syslog-ng-mod-redis depends on syslog-ng-core (>= 3.27.1-2); however:
  Package syslog-ng-core is not configured yet.
 syslog-ng-mod-redis depends on syslog-ng-core (<< 3.27.1-2.1~); however:
  Package syslog-ng-core is not configured yet.

Fix:

Backup configuration:

mkdir ~/syslog-ng_backup/
cp -rf /etc/syslog-ng/* ~/syslog-ng_backup/

Verify configuration:

ls ~/syslog-ng_backup/

Purge syslog-ng and remove everything:

sudo apt purge syslog-ng-core

If some files remain, delete them all:

rm -rf /etc/syslog-ng

Reinstall syslog-ng-core:

sudo apt install syslog-ng-core

Reinstall syslog-ng:

sudo apt install syslog-ng

Cleanup some packages:

sudo apt auto-remove

Restore RS configuration files:

cp ~/syslog-ng_backup/conf.d/99* /etc/syslog-ng/conf.d/

If you edited the /etc/syslog-ng/syslog-ng.conf file, check the difference and restore your custom configuration.

This issue should be fixed in version 3.27.1-2.1 and higher.

12.6 How to resolve a full disk¶

Error recieved in Kibana:

{"type":"log","@timestamp":"2022-08-29T15:55:49+02:00","tags":["info","savedobjects-service"],"pid":8508,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 64ms."}
{"type":"log","@timestamp":"2022-08-29T15:55:49+02:00","tags":["info","savedobjects-service"],"pid":8508,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_TRANSFORM -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK. took: 124ms."}

Run:

curl -XGET -H "Content-Type: application/json" http://elastic:elastic@localhost:9200/_cluster/allocation/explain?pretty

Given warning:

"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [13.201362304896152%]"

Temporary increase the allowed diskspace:

curl -XPUT -H "Content-Type: application/json" http://elastic:elastic@localhost:9200/_cluster/settings -d '
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "92%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}' | jq

Login and remove some shards to lower the diskspace then restore the allowed diskspace:

curl -XPUT -H "Content-Type: application/json" http://elastic:elastic@localhost:9200/_cluster/settings -d '
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}' | jq