Capturing, Monitoring, Alerting, Logging & Visualizing with ELK stack
Elasticsearch, Logstash, and Kibana being an open-source product provide a way to collate data from different distributed sources, process according to the needs, and visualize the same in many different types of charts, plotting it into dashboards.
It performs ETL sort of work.
It can capture application-level data, system-level, as well as network-level data, and the list can go on and on.
It comprises many components for various purposes
Application Level Data
1. Capturing all web requests:
The stack provides an addon for Application Performance Monitoring in form of APM Server and APM Agents which are in various languages. Here we will use java APM agents
a. APM Server
Below is the Docker deployment script snippet of APM Server
apm-server:
container_name: apm-server
hostname: apm-server
image: "docker.elastic.co/apm/apm-server:${VERSION}"
restart: on-failure
volumes:
- ./init.sh:/usr/share/apm-server/init.sh
command: bash -c "init.sh && apm-server -e -E output.elasticsearch.hosts=["${ELASTICSEARCH_HOST}:9200"] -E apm-server.host=0.0.0.0:8200 -E setup.kibana.host=${KIBANA_HOST}:5601 -setup -strict.perms=false"
depends_on:
- elasticsearch
- kibana
ports:
- "8200:8200"
environment:
- ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}
- KIBANA_HOST=${KIBANA_HOST}
b. APM Agent
APM agents instrument the application captures the data and passes the same to the APM server. We need to configure APM SERVER's URL and Port and it gets connected to the APM server and starts passing the data. This data is of all web requests/responses, caching data, and data generated while connecting to external URLs.
Set the below configuration in the VM arguments or set it as a system property
To enable APM capturing in Wildfly Configuration
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"
set "JAVA_OPTS=%JAVA_OPTS% -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/a:%JBOSS_HOME%/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.7.Final.jar -Xbootclasspath/a:%JBOSS_HOME%/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.4.Final.jar"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.server_url=http://localhost:8200"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.application_packages=com.de"
set "JAVA_OPTS=%JAVA_OPTS% -javaagent:D:/salm/jboss/wildfly11-adfs/ExportedFiles/elastic-apm-agent-0.6.0.jar"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.service_name=wildfly"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.capture_body=all"
To enable APM capturing in JBoss Configuration
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.server_url=http://localhost:8200"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.application_packages=com.de"
set "JAVA_OPTS=%JAVA_OPTS% -javaagent:D:/salm/jboss/wildfly11-adfs/ExportedFiles/elastic-apm-agent-0.6.0.jar"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.service_name=perf"
set "JAVA_OPTS=%JAVA_OPTS% -Delastic.apm.capture_body=all"
Note: This has a performance impact on the application so check it before deploying on production
2. Capturing all MBeans:
Steps :
1. Deploy the Jolokia war file on the application server
Jolokia.war path https://jolokia.org/download.html
This allows to expose all mbeans if they are not restricted on the application server over HTTP.
2. Capture the exposed mbeans and push them into elasticsearch
This can be achieved using metricbeat. The following allows capturing memory, operation system, database, and jms queue metadata.
Below is the script snippet
module: jolokia
metricsets: ["jmx"]
period: 10s
hosts: ["${APP_SERVER1}"]
namespace: "metrics"
jmx.mappings:
- mbean: 'java.lang:type=Runtime'
attributes:
- attr: Uptime
field: uptime
- mbean: 'java.lang:type=Memory'
attributes:
- attr: HeapMemoryUsage
field: memory.heap_usage
- attr: NonHeapMemoryUsage
field: memory.non_heap_usage
- mbean: 'java.lang:type=Threading'
attributes:
- attr: ThreadCount
field: thread.count
- attr: DaemonThreadCount
field: thread.daemon
- mbean: 'java.lang:type=OperatingSystem'
attributes:
- attr: FreePhysicalMemorySize
field: cpu.freephysicalmemorysize
field_type: integer
- attr: AvailableProcessors
field: cpu.availableprocessors
field_type: integer
- attr: SystemLoadAverage
field: cpu.systemloadaverage
field_type: integer
- attr: TotalPhysicalMemorySize
field: cpu.totalphysicalmemorysize
field_type: integer
- attr: ProcessCpuLoad
field: jvm.process_cpu_load
# Wildfly
#- mbean: 'jboss.as:data-source="java:/JNDI_DIGITEDS",statistics=pool,subsystem=datasources'
# attributes:
# - attr: MaxUsedCount
# field: datasource.maxusedcount
# - attr: AvailableCount
# field: datasource.availablecount
# - attr: InUseCount
# field: datasource.inusecount
- mbean: 'jboss.jca:name=JNDI_DS,service=ManagedConnectionPool'
attributes:
- attr: AvailableConnectionCount
field: datasource.availableconnectioncount
field_type: integer
- attr: ConnectionCount
field: datasource.connectioncount
field_type: integer
- attr: ConnectionCreatedCount
field: datasource.connectioncreatedcount
field_type: integer
- attr: ConnectionDestroyedCount
field: datasource.connectiondestroyedcount
field_type: integer
- attr: InUseConnectionCount
field: datasource.inuseconnectioncount
field_type: integer
- attr: MaxConnectionsInUseCount
field: datasource.maxconnectionsinusecount
field_type: integer
- mbean: 'org.hornetq:module=JMS,name="JNDI_QUEUE",type=Queue'
attributes:
- attr: MessageCount
field: queue.projectcreationcheckinqueue.messagecount
field_type: integer
- attr: MessagesAdded
field: queue.projectcreationcheckinqueue.messagesadded
field_type: integer
- attr: ConsumerCount
field: queue.projectcreationcheckinqueue.consumercount
field_type: integer
Troubleshooting
Kibana Troubleshooting :
Kibana Troubleshooting
Logstash Troubleshooting :
Push MongoDB data to ElasticSearch through Logstash
Logstash Troubleshooting
Comments
Post a Comment