Elasticsearch is a distributed, open-source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elastic.
At Forem we use Elasticsearch for all of our user-facing searching needs. Models that are searched using Elasticsearch are:
The above models are organized in Elasticsearch in different indexes to make searching more accurate and performant. The index breakdown is:
You will notice that Podcast Episodes, Comments, and Articles all share the same index. This allows us to search and aggregate across all of these documents at the same time. These documents also share a lot of similar fields so putting them in the same index made sense. In the future, depending on our search needs this index structure might change.
Each index that is listed above has a corresponding mapping that defines what fields can and cannot be indexed to it. These mappings are similar to a db schema. In our case, all of our index mappings are defined as strict
which means if we try to index a field that is not defined it will throw an error. Mappings for each index can be found in app/config/elasticsearch/mappings
.
In order to index the data from Postgres into Elasticsearch, we rely on callbacks in our ActiveRecord models. Whenever a model is created or updated we use an after_commit
callback to enqueue a Search::IndexWorker
which handles indexing the document into Elasticsearch. In order to translate our ActiveRecord data to Elasticsearch we use serializers. Each model has its own serializer which can be found in app/serializers/search
Once the documents are in Elasticsearch then we can search for them. This is handled by code in our app/services/search
directory. Here you will see a collection of classes used to help us index and search documents in Elasticsearch. The nested query_builders
directory contains all the logic we need to help us build complex search queries to send to Elasticsearch.
When adding a new field to Elasticsearch here are the steps you have to follow:
app/config/elasticsearch/mappings
. If you plan to test locally you will need to run bin/setup
to update your local Elasticsearch mappings.app/serializers/search
.