Note that these documents have some common fields, notably, host.env, the name of the environment where the service is running. On the left, we see the JSON documents transformed by Logstash from Jenkins and Nginx production logs. The core idea is to group services by environment and write their logs to the same index alias. S1: Group by environment - rotate by age and size A quality strategy is easy to implement and maintain, adapt to changes to the service landscape, and efficiently use the Elasticsearch resources (disk, CPU, memory). In the context of this blog post, a log indexing strategy is defined by (i) an index name pattern, (ii) an alias for all the indices matching the same name pattern (also called backing indices), and (iii) a log rotation policy. Finally, the Logstash pipelines use the Elasticsearch output plugin to ship the processed logs as JSON documents to the specified indices in the Elasticsearch cluster. In particular, every processed log event will have a field for the service name and another for the environment. We set up Filebeats shippers to send log streams over to a Logstash server where we configured pipelines to normalize the diverse log formats to a more consistent structure to enhance correlation (see the Elastic Common Schema for inspiration). We chose the Elastic Stack to centralize logging of a heterogeneous group of services running on multiple environments. As we will see, none of these strategies will be entirely satisfactory, but they will build the case for a third log indexing strategy… But that I’ll leave as a cliffhanger for a second, conclusive blog post. This blog post will present two log indexing strategies and evaluate them based on maintainability and efficiency criteria.
For example, how can we enforce different retention policies for dozens of log sources without getting into configuration hell? How flexible is our strategy to accommodate new log sources with the least amount of effort? What number and size of indices will optimize ingestion throughput, disk utilisation and analysis performance? We will get into the shoes of the DevOps team in our use case scenario and uncover the trade-offs between resource efficiency, reliability, and maintainability when choosing a log indexing strategy. This time, I’d like to step back in the ingestion process and address related, but distinct challenges, focusing on how we create and configure the Elasticsearch indices for our logs. Issues like dealing with missing log entries, bad search performance and usability.
#Filebeats to elastisearch series
In a previous series of blog posts, we have already talked about common issues - and possible solutions - when analyzing logs in Kibana. So, that’s it, right? What can go wrong? Plenty. Then, Logstash indexes (persists) the processed data in the Elasticsearch search and analytics engine, which your team can query using the Kibana UI. As illustrated below, Beats modules can collect logs and metrics on the hosts and ship them to Logstash where they are parsed, sanitized and enriched.
The Elastic Stack offers a unifying solution for all these activities. Part of your daily job is to keep a close eye on these services, monitoring their performance, detecting anomalies, and doing a speedy root cause analysis when everything’s on fire. These services can run on multiple environments, and their number and type can change over time (e.g., by adding contract testing integration to your CI/CDs). Think of the version control system, CI/CD tools, deployment environments, and so on. Say you operate the shared DevOps platform that enables developers in your company to roll out new features and hotfixes. Imagine your team is responsible for business-critical IT services.
0 kommentar(er)
