Logstash Hardware Requirements

Log processing at Scale. ELK cluster at 25k events per second

Beats and Logstash make ingest awesome. Together, they provide a comprehensive solution that is scalable and resilient. What can you expect?

  • Horizontal scalability, high availability, and variable load handling
  • Message durability with at-least-once delivery guarantees
  • End-to-end secure transport with authentication and wire encryption

Beats run across thousands of edge host servers, collecting, tailing, and shipping logs to Logstash. Logstash serves as the centralized streaming engine for data unification and enrichment. The Beats input plugin exposes a secure, acknowledgement-based endpoint for Beats to send data to Logstash.

Enabling persistent queues is strongly recommended, and these architecture characteristics assume that they are enabled. We encourage you to review the Persistent Queues documentation for feature benefits and more details on resiliency.

Logstash is horizontally scalable and can form groups of nodes running the same pipeline. Logstash’s adaptive buffering capabilities will facilitate smooth streaming even through variable throughput loads. If the Logstash layer becomes an ingestion bottleneck, simply add more nodes to scale out. Here are a few general recommendations:

  • A minimum of two Logstash nodes are recommended for high availability.
  • It’s common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources.

When using Filebeat or Winlogbeat for log collection within this ingest flow, at-least-once delivery is guaranteed. Both the communication protocols, from Filebeat or Winlogbeat to Logstash, and from Logstash to Elasticsearch, are synchronous and support acknowledgements. The other Beats don’t yet have support for acknowledgements.

Logstash persistent queues provide protection across node failures. For disk-level resiliency in Logstash, it’s important to ensure disk redundancy. For on-premise deployments, it’s recommended that you configure RAID. When running in the cloud or a containerized environment, it’s recommended that you use persistent disks with replication strategies that reflect your data SLAs.

Make sure queue.checkpoint.writes: 1 is set for at-least-once guarantees. For more details, see the persistent queue durability documentation.

Logstash will commonly extract fields with grok or dissect, augment geographical info, and can further enrich events with file, database, or Elasticsearch lookup datasets. Be aware that processing complexity can affect overall throughput and CPU utilization. Make sure to check out the other available filter plugins.

Enterprise-grade security is available across the entire delivery chain.

  • Wire encryption is recommended for both the transport from Beats to Logstash and from Logstash to Elasticsearch.
  • There’s a wealth of security options when communicating with Elasticsearch including basic authentication, TLS, PKI, LDAP, AD, and other custom realms. To enable Elasticsearch security, consult the X-Pack documentation.

When running Logstash 5.2 or greater, the Monitoring UI provides deep visibility into your deployment metrics, helping observe performance and alleviate bottlenecks as you scale. Monitoring is an X-Pack feature under the Basic License and is therefore free to use. To get started, consult the X-Pack Monitoring documentation.

Users may have other mechanisms of collecting logging data, and it’s easy to integrate and centralize them into the Elastic Stack. Let’s walk through a few scenarios:

The TCP, UDP, and HTTP protocols are common ways to feed data into Logstash. Logstash can expose endpoint listeners with the respective TCP, UDP, and HTTP input plugins. The data sources enumerated below are typically ingested through one of these three protocols.

The TCP protocol does not support application-level acknowledgements, so connectivity issues may result in data loss.

For high availability scenarios, a third-party hardware or software load balancer, like HAProxy, should be added to fan out traffic to a group of Logstash nodes.

Although Beats may already satisfy your data ingest use case, network and security datasets come in a variety of forms. Let’s touch on a few other ingestion points.

  • CEF - Logstash accepts and parses CEF data from systems like Arcsight SmartConnectors with the CEF codec. See this blog series for more details.

Existing syslog server technologies like rsyslog and syslog-ng generally send syslog over to Logstash TCP or UDP endpoints for extraction, processing, and persistence. If the data format conforms to RFC3164, it can be fed directly to the Logstash syslog input.

Blue Sea Systems Blue Sea Systems 250V AC LED, Red
Sports (Blue Sea Systems)
  • Easily installed in any Blue Sea Systems circuit breaker panel
  • Simple push-in installation mounts in any thickness material
  • Useful as general indicator and alarm lights
  • 26 AWG and 11/64 inch mounting hole size
Acr Electronics Blue Sea Systems 5025 ST Blade Fuse Block - 6 Circuits with Negative Bus and Cover
Sports (Acr Electronics)
  • St blade fuse block - 6 circuits with negative bus and cover
  • Positive distribution bus with #10-32 stud
  • Can be used for 24-hour circuits
  • Cover satisfies ABYC/USCG Requirements for insulation, incorporates an easy to open push button latch providing easy access to fuses, storage for two spare fuses
  • Accepts ATO and ATC fast acting blade fuses
Morsen Morsen 1200W Full Spectrum COB LED Grow Light System Panel Lamp Indoor Flower Veg Plant Yard Garden Replace HPS
Lawn & Patio (Morsen)
  • Actual power comsumption 200W, brig the brightness as1200W HPS, big advantage of led light is power saving save about 50% power consumption of the light
  • COB(Chips on Board) integrated chip is the latest technology of LED packaging for LED engine; Integrated COB led grow lights improving color uniformity and luminous...
  • Unique design of reflective cup for higher light-gathering effect. 2 on/off switch, flexible to control the light in half and full running.
  • Color Ratio:Full Spectrum 380nm-840nm.All color plant need, and suit for all stage.
  • Daisy-chain connection is available. 2-3 lights can be connected and used together by power cords. Energy & money & time saving. Power cord is included in...
Fit System Fit System 70119T Toyota FJ Cruiser Passenger Side Replacement Convex Mirror with Lamp
Automotive Parts and Accessories (Fit System)
  • Passenger side mirror, tested to fit and function like the original, meets or exceeds OEM standards.
  • Heated and power adjustable, OE-comparable wiring harness (no pigtail connector) for hassle-free installation.
  • Can fold mirror to park in narrow garages
  • Paint-To-Match, mirror is painted in black but is paintable to match color of the car.
  • Wide-angle lens for maximum visibility with warning on the glass to meet FMVSS requirement.
Mingdak Mingdak® LED Aquarium Light Fixture for Fish Tanks,suitable for Saltwater and Freshwater,72 Leds,20-inch,lighting Color White and Blue
Pet Products (Mingdak)
  • Super bright, energy efficient and long lasting LEDs Brighter than one fluorescent tube fixture
  • Aasy-to-install LED light fixture with adjustable mounting legs which fit your aquarium tank from 20 inches to 27 inches
  • Two lighting modes:White & Blue light on for the daylight and only blue light on for the nightlight
  • Meets most of the basic requirements of a freshwater/saltwater tank for purpose of keeping fish happy
  • Provide the ultimate opportunity to create a realistic underwater home-aquarium environment

Related posts: