Elasticsearch Service + Python 연동

1. 코드 세팅

• application.py

import boto3
from flask import Flask, request, jsonify
from flask_cors import CORS
import os
from flaskext.mysql import MySQL
import redis
import logging
from logstash_async.handler import AsynchronousLogstashHandler

application = Flask(__name__)

# cors
cors = CORS(application, resources={r"/*": {"origins": "*"}})

# mysql
mysql = MySQL()
application.config['MYSQL_DATABASE_USER'] = os.environ["MYSQL_DATABASE_USER"]
application.config['MYSQL_DATABASE_PASSWORD'] = os.environ["MYSQL_DATABASE_PASSWORD"]
application.config['MYSQL_DATABASE_DB'] = os.environ["MYSQL_DATABASE_DB"]
application.config['MYSQL_DATABASE_HOST'] = os.environ["MYSQL_DATABASE_HOST"]
mysql.init_app(application)

# redis
db = redis.Redis(os.environ["REDIS_HOST"], decode_responses=True)

#logstash
python_logger = logging.getLogger('python-logstash-logger')
python_logger.setLevel(logging.INFO)
python_logger.addHandler(AsynchronousLogstashHandler(os.environ["LOGSTASH_HOST"], 5044, database_path=''))

@application.route('/')
def main():
    python_logger.info('main')
    return "핵심 쏙쏙 AWS"

@application.route('/fileupload', methods=['POST'])
def file_upload():
    file = request.files['file']
    s3 = boto3.client('s3',
                      aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
                      aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"]
                      )
    s3.put_object(
        ACL="public-read",
        Bucket=os.environ["BUCKET_NAME"],
        Body=file,
        Key=file.filename,
        ContentType=file.content_type
    )

    conn = mysql.connect()
    cursor = conn.cursor()
    cursor.execute("insert into file(file_name) value('" + file.filename + "')")
    conn.commit()

    cursor.execute("SELECT count(*) from file")
    data = cursor.fetchone()
    conn.close()
    db.set("fileCount", data[0])

    python_logger.info(file.filename)
    return jsonify({'result': 'success'})

@application.route('/files', methods=['GET'])
def files():
    conn = mysql.connect()
    cursor = conn.cursor()
    cursor.execute("SELECT file_name from file")
    data = cursor.fetchall()
    conn.close()

    return jsonify({'result': 'success', 'files':data})

@application.route('/file/count', methods=['GET'])
def file_count():
    return jsonify({'result': 'success', 'count':db.get("fileCount")})

if __name__ == '__main__':
    application.debug = True
    application.run()

 

2. 라이브러리 추가 (파이참/로컬)

#터미널
pip install python-logstash-async

or 파이참에 인터프리터 설정
python-logstash-async 검색후 설치

 

3. requirements.txt 업데이트

pip freeze > requirements.txt

 

4. EC2에 5044 포트 열어주기

• 보안 -> 보안그룹 -> 인바운드 규칙 편집 -> 추가 : 포트 5044

 

5. EB 환경변수 추가

• LOGSTASH_HOST 추가  : EC2 인스턴스의 퍼블릭 IPv4

 

6. commit -> push

 

7. kibana 접속-> 메뉴 -> stack management -> Index Patterns

 

Create index pattern 클릭 하여 생성

8. kibana 메뉴 -> Discover

 

 

** elasticsearch 가격테이블 : https://aws.amazon.com/ko/elasticsearch-service/pricing/

** elasticsearch 가격 계산 : https://calculator.aws/#/createCalculator/ElasticsearchService

 

elasticsearch는 비싸다