[elk] logstash imap 플러그인 사용하기

logstash imap 플러그인 사용하기

imap 이란 Internet Mail Access Protocol 의 약자로 인터넷 메일에 접근하기위한 프로토콜이라할수 있다.

imap 입력 플러그인을 사용해서 수신된 메일을 elasricsearch 에 인덱싱하기

1. email.conf 생성
   이전 포스팅과 마찬가지로 /usr/share/logstash/files 디렉토리에 생성

cd /usr/share/logstash/files
vi email.conf

email.conf 내용

input {
  imap {
    host => "imap.daum.net"
    password => "<이메일비밀번호>"
    user => "<이메일계정>"
    port => 993
    check_interval => 10
    folder => "Inbox"
    secure => true
  }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    index => "emails"
    document_type => "email"
    hosts => ["localhost:9200"]
    user => "<elasticsearch 계정>"
    password => "<elasticsearch 비밀번호>"
  }
}

나는 다음메일을 사용하기 때문에 일단 브라우저로 메일에 접속해서 imap 설정을 해줘야한다.

환결설정 > IMAP/POP3 페이지에서 IMAP/SMTP 사용 을 사용함으로 변경하고 IMAP 서버명, 포트번호, 아이디, 비밀번호를 email.conf 파일에 기재한다.

2. 실행

cd /usr/share/logstash
bin/logstash -f file/email.conf

[INFO ] 2021-01-15 11:25:05.869 [[main]<imap] imap - Saving "uid_last_value": "360"
{
                    "subject" => "오랜만입니다",
        "x-hermes-message-id" => "p0FBAvesF548683427",
                 "x-received" => "by 2002:a67:fd67:: with SMTP id h7mr8982218vsa.9.1610676656592; Thu, 14 Jan 2021 18:10:56 -0800 (PST)",
                       "date" => "Thu, 14 Jan 2021 18:10:40 -0800",
               "received-spf" => "none (mx.daum.net: domain of ulronaldobks@ngerjiwon.silok.513fy.com does not designate permitted sender hosts)",
         "x-hanmail-env-from" => "ulronaldobks@ngerjiwon.silok.513fy.com",
    "x-google-dkim-signature" => "v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=yojMGeNTzQpHy7cnDboWXrEYEcO/0p4OYmZ+jk69XmQ=; b=iNqB6TkRFhy8Rf0a6ERLfd/lmLpkyeYELZ02b1th8ZTlAuEy5crHu5YksPcctQcCY7 0ZrqoE6jhQZGWo2eqsMEsXhwsOEoiGlSDJGt4zHSU0fkoUfmzSSPH2Z9G11hPLqKfKAh 8b4e+S0SIjmyLKWJHmpZVR9vOH9b6MqOpLftsLLOlEX6PlAUPrR9TEZIawOKFmhBBfS/ 7/erIuFNzGgjCT91etV3a7aj1vtQ7i4dUczQetAK0mqxZskMC6j90SFJ9Z+g1+XwF5nf exza24psrMONh6x+ox6NYg37af0Dpvt2jg9TdGwwjzCkE79U0XC9TksIewlwGgXxOb9e VkuQ==",
           "x-originating-ip" => "[103.122.177.82]",
               "mime-version" => "1.0",
            "x-hanmail-class" => "W",
         "x-gm-message-state" => "AOAM532HL9yiadQ0v/TYLJhQbWHsqJuiPJbfETXIAGLqVQWHtQUVKTuf 947GkSab9gHYgBSNq2Y+trAcfpJdSTPa2kBFlyLbRa7TbGjI+b6IwO08bA==",
               "content-type" => "multipart/alternative; boundary=000000000000f6dc5705b8e6e12b",
                         "to" => "nerobong3@daum.net",
                   "received" => [
        [0] "from mail-vs1-f66.google.com ([209.85.217.66]) by hermes of mail-rmail26 (10.194.27.157) with ESMTP id p0FBAvesF548683427 for <nerobong3@daum.net> (version=TLSv1.2 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 15 Jan 2021 11:10:57 +0900",
        [1] "by mail-vs1-f66.google.com with SMTP id r24so4233898vsg.10 for <nerobong3@daum.net>; Thu, 14 Jan 2021 18:10:57 -0800",
        [2] "by 2002:ab0:28f:0:0:0:0:0 with HTTP; Thu, 14 Jan 2021 18:10:40 -0800 (PST)"
    ],
                   "@version" => "1",
                 "message-id" => "<CAMzS3jHHit+Kz8amh2z1+dZRW6bc8f2qKRWa8ae86otE8z5YLw@mail.gmail.com>",
                       "from" => "\"조평제\" <ulronaldobks@ngerjiwon.silok.513fy.com>",
       "x-google-smtp-source" => "ABdhPJwEgLN7jEtfombokq4ujTopL0nhFRjJ5mXCHM01pGYWOCSvm4mfKW57BAPYQNuYROcFPaHBElsX9G8rapUiamE=",
          "x-hanmail-peer-ip" => "209.85.217.66",
                 "@timestamp" => 2021-01-15T02:10:40.000Z,
             "dkim-signature" => "v=1; a=rsa-sha256; c=relaxed/relaxed; d=ngerjiwon-silok-513fy-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=yojMGeNTzQpHy7cnDboWXrEYEcO/0p4OYmZ+jk69XmQ=; b=1UYV0GyYwkMS+xBH/DPh/TVlpg5PbfeQAja4/NTYoEJhg+2EsA68lqy1q55Eam86AG lwWmCIpIG97hWwefLg7Wt/Uhl8OKMFZ3bnsYUFQb5aVeDSB3BcIUdgf7wKJ5PVUvXQaL lgf1CQ9eSvcLRjGp5F1c7l3X8JEJ+8r4IuXgGBdk8R4BABrz2FozhP9qXz2umU9F4oQM NJM3LWmlaopi1hpglxJgVZWsFEcwmVpxKQ27q2PcwjHJlnhOqhuw0drByDfQ5hI+sEBL x509MH8q3RcMebUOudMEPoAeSYdxahtAKUCo4W1cihkefDLjdnF6MK7SYosuAEHcY8CO GXaQ==",
                    "message" => "쉿! 비아그라, 시알리스, 조루 방지제!\r\n\r\n여성흥분제! 최음제■비밀 배송■\r\nhttp://hlv.gt585.com/fnerobong3.0110\r\n"
}
[INFO ] 2021-01-15 11:25:13.924 [[main]<imap] imap - Saving "uid_last_value": "360"
[INFO ] 2021-01-15 11:25:23.837 [[main]<imap] imap - Saving "uid_last_value": "360"
[INFO ] 2021-01-15 11:25:33.851 [[main]<imap] imap - Saving "uid_last_value": "360"
[INFO ] 2021-01-15 11:25:44.625 [[main]<imap] imap - Saving "uid_last_value": "360"

그새 무슨 이상한 메일이 하나 와서 로그에 찍힘.

3. elasricsearch 데이터 확인

GET /emails/_search
{
  "query": {"match_all": {}}
}

{
  "took" : 805,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "emails",
        "_type" : "email",
        "_id" : "R7esA3cB-Q87Y8iFWfXv",
        "_score" : 1.0,
        "_source" : {
          "to" : "nerobong3@alancorp.net",
          "date" : "Fri, 15 Jan 2021 10:32:15 +0900",
          "received-spf" : "pass (mx.daum.net: domain of nerobong2@gmail.com designates 209.85.210.46 as permitted sender)",
          "x-hanmail-peer-ip" : "209.85.210.46",
          "message-id" : "<CAOLNi-2+XWgb=uA+Cy+XeSdfQV8yczG5-Vdzt7hX0VkcFe8pDg@mail.gmail.com>",
          "x-hermes-message-id" : "p0FAWS2q11877401274",
          "received" : [
            "from mail-ot1-f46.google.com ([209.85.210.46]) by hermes of mail-kairos-rmail5 (10.194.27.126) with ESMTP id p0FAWS2q11877401274 for <nerobong3@alancorp.net> (version=TLSv1.2 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 15 Jan 2021 10:32:28 +0900",
            "by mail-ot1-f46.google.com with SMTP id w3so7080890otp.13 for <nerobong3@alancorp.net>; Thu, 14 Jan 2021 17:32:27 -0800"
.
.
.
.
.

잘들어갓다.~