Amazon OpenSearch is an open-source, distributed search and analytics engine built on top of the popular Elasticsearch technology. It allows users to store, search, and analyze large volumes of data in near real-time, making it an ideal solution for applications that require fast and reliable search capabilities.
Elastic NV changed the licensing terms which brings limitation and not open source. So to avoid vendor lock-in, Amazon has created a fork from last ALv2 version of ElasticSearch and Kibana. This fork has been named as OpenSearch. Amazon OpenSearch is designed to be highly scalable and flexible, allowing users to deploy and manage their search clusters in the cloud, on-premises, or in hybrid environments. It includes a wide range of features and tools, including advanced search capabilities, analytics, monitoring, and security features, which can be customized to meet specific business needs.
Use Cases:
- Log Analytics
- Real time application monitoring
- Security Analytics
- Full text Search
- ClickStream Analytics
- Indexing
- Anomaly detection
OpenSearch Components:
- OpenSearch (Forked from ElasticSearch project ALv2) provides search and indexing capability
- OpenSearch Dashboard(Forked from Kibana Project ALv2) – Provides real-time dashboards on top of the data that is available in OpenSearch. It’s also an alternative to CloudWatch dashboard
- Logstash – Log ingestion mechanism, use the “Logstash Agent”
Can we use OpenTelemetry for log ingestion?
OpenTelemetry for creation, propagation, collection, processing, and exporting of trace data. OpenTelemetry is an open-source project that provides a set of APIs, libraries, agents, and other tools to enable observability in cloud-native and distributed systems. It is designed to standardize the collection, aggregation, and export of telemetry data, including traces, metrics, and logs, from applications and infrastructure components across different programming languages and platforms.
OpenTelemetry was formed by merging the OpenTracing and OpenCensus projects, and it is currently maintained by the Cloud Native Computing Foundation (CNCF). Its main goal is to make it easier for developers and operators to instrument their applications and services for monitoring and troubleshooting, without requiring them to write custom code or rely on proprietary solutions.
OpenTelemetry metrics can be exported to OpenSearch.
How to Migrate from ElasticSearch?
Amazon OpenSearch is fully compatible with the Elasticsearch API, which means that users can easily migrate their existing Elasticsearch workloads to Amazon OpenSearch without having to modify their applications or APIs. It is also integrated with other Amazon Web Services (AWS) tools and services, such as Amazon S3, Amazon Kinesis, and Amazon CloudWatch, making it easy to build and deploy complex search and analytics workflows.
How to launch OpenSearch in AWS ?
AWS offers managed cluster and Serverless. In Managed cluster, You need to provide the instance type and number of data nodes for the OpenSearch cluster.
The below steps will guide to create the managed OpenSearch cluster in AWS.
- Create an Amazon VPC (Virtual Private Cloud): OpenSearch requires a VPC to operate. If you already have one, you can skip this step
- Launch an OpenSearch cluster: Navigate to the OpenSearch service page in the AWS Management Console and click “Create a domain”. Give your domain a name, choose the OpenSearch version and node configuration that best suits your needs, and select your VPC. You can also configure advanced settings, such as encryption and access policies, in this step
- Configure your cluster: After your cluster is created, you can configure it by adding indices and configuring access policies, among other tasks
- Access your cluster: Once your cluster is up and running, you can access it using the endpoint URL provided by AWS. You can also use Kibana, an open-source data visualization tool, to interact with your data
Please note that there may be additional steps or variations depending on your specific use case and requirements. I recommend consulting the OpenSearch documentation and AWS support resources for more information.
In Serverless option, you just need to create collections to start using the service.
Conclusion
In summary, Elasticsearch and OpenSearch are similar in many ways, but differ in ownership, licensing, governance, and development model, as well as in some technical features. The choice between the two depends on your specific use case and requirements. Elasticsearch is developed by Elastic, with contributions from a community of developers, and its roadmap is largely controlled by Elastic. OpenSearch, on the other hand, is developed and governed by the OpenSearch Foundation, a non-profit organization that includes members from several companies, and its roadmap is driven by the community.
There are also some technical differences between the two. OpenSearch includes some features that are not available in Elasticsearch, such as anomaly detection and alerting. OpenSearch also includes some performance improvements and bug fixes that are not yet available in Elasticsearch.
