/ mozey.co / blog

elastic search

October 24, 2016

🔗 Elasticsearch 5.x

Install

mkdir /opt/elasticsearch

cd /opt/elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.3.tar.gz

sudo tar -xvzf elasticsearch-5.6.3.tar.gz

sudo chown -R USER elasticsearch-5.6.3

Run

cd elasticsearch-5.6.3

./bin/elasticsearch

Check it’s working

http 'http://localhost:9200'

🔗 Kibana 5.x

🔗 Install

cd /opt/elasticsearch

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.3-linux-x86_64.tar.gz

sha1sum kibana-5.6.3-linux-x86_64.tar.gz 

sudo tar -xzf kibana-5.6.3-linux-x86_64.tar.gz

Run

cd kibana-5.6.3-linux-x86_64/

./bin/kibana

Config

sudo vi config/kibana.yml 
# server.host: "MY.IP.ADDRESS"

Check it’s working

http 'http://localhost:9200'

🔗 Getting started

Download Shakespeare data set

wget https://download.elastic.co/demos/kibana/gettingstarted/shakespeare.json

wget https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip

wget https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz

unzip accounts.zip

gunzip logs.jsonl.gz

Setup mapping for the Shakespeare and logs data sets

curl -XPUT 'localhost:9200/shakespeare?pretty' -H 'Content-Type: application/json' -d'
{
 "mappings" : {
  "_default_" : {
   "properties" : {
    "speaker" : {"type": "keyword" },
    "play_name" : {"type": "keyword" },
    "line_id" : { "type" : "integer" },
    "speech_number" : { "type" : "integer" }
   }
  }
 }
}
'

curl -XPUT 'localhost:9200/logstash-2015.05.20?pretty' -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "log": {
      "properties": {
        "geo": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}
'

Bulk load into elasticsearch

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/_bulk?pretty' --data-binary @shakespeare.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl

Verify bulk load

curl 'localhost:9200/_cat/indices?v'

🔗 Filebeat

Install

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.3-amd64.deb

sudo dpkg -i filebeat-5.6.3-amd64.deb

Config

sudo vi /etc/filebeat/filebeat.yml 

Test config

sudo /usr/bin/filebeat.sh -configtest -e

Manually load template

curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/filebeat' [email protected]/etc/filebeat/filebeat.template.json

Service

sudo /etc/init.d/filebeat start

sudo /etc/init.d/filebeat stop

🔗 Registry file

Delete filebeat index

curl -XDELETE 'http://localhost:9200/filebeat-2017.10.25'

Clear registry

sudo vi /var/lib/filebeat/registry

Run filebeat only once until all harvesters reach EOF

sudo /usr/bin/filebeat.sh -once

Run filebeat with modules enabled

cd /opt/elasticsearch/elasticsearch-5.6.3

sudo bin/elasticsearch-plugin install ingest-geoip

sudo /usr/bin/filebeat.sh -e -modules=system -setup

🔗 Apache2 Module

TODO

🔗 ELK Stack Arduino