A search engine for hundreds of thousands of multilingual contents: Meilisearch

Companies with large content platforms face enormous challenges when it comes to implementing an efficient search solution. This is particularly true for projects like TutKit.com, a comprehensive learning and content platform that provides content in multiple languages, where search becomes a central element. The expectations are high: a fast, efficient, and multilingual search solution is crucial to provide users with an optimal experience.

Our platform, TutKit.com, offers tens of thousands of content items, each available in dozens of languages. This variety and volume of content place significant demands on the search function, along with additional challenges regarding performance, database load, and database security. The search must not only be fast and accurate but also effectively support multilingualism. Here is a case study from us on how we found the perfect search engine for us with Meilisearch.

The perfect search engine for content platforms: challenges and expectations

At the beginning, we had a search solution implemented by an external agency. The implemented search was based on MySQL and was, to put it mildly, inadequate. The loading times for search results took up to 10 seconds, which was frustrating for our users and severely impacted the user experience. Additionally, this solution could not provide good multilingual support. This external agency, although specialized in Laravel, was simply overwhelmed by a project like TutKit.com. It was clear: a drastic improvement was necessary. And we would handle it ourselves.

The decision for Meilisearch

In search of a better solution, we came across Meilisearch. What particularly impressed us about Meilisearch was its speed and modern user experience. The web presence of Meilisearch and the compelling UX were an initial confirmation that the company is modern and agile. The real excitement came when we tested the speed of the search results. It was a difference like night and day compared to our previous solution. Additionally, Meilisearch offered excellent multilingual support, which was crucial for us.

1000 results in 2 ms… that’s quite a statement:

We ultimately chose the open-source version of Meilisearch. This version was selected because it gave us complete control over the search results, without additional costs for licensing fees or external developers, where it is not even certain if they would be able to manage the project, as we have unfortunately experienced in the past.

The implementation of Meilisearch

The integration of Meilisearch into our platform TutKit.com was the first step towards improving the search functionality. Here are the details of how we implemented Meilisearch and how it is currently managed in our company.

Setup and configuration

We decided to use Meilisearch in its open-source version to maintain complete control over our search solutions. The installation began by downloading the Meilisearch server from the official website. To ensure stable and reliable operation, we set up a systemd service. This service was configured to automatically start Meilisearch as soon as the server boots up and to automatically restart the service in case of a failure.

# Beispiel für eine systemd-Dienstdatei für Meilisearch
[Unit]
Description=MeiliSearch
After=systemd-user-sessions.service

[Service]
Type=simple
ExecStart=/usr/bin/meilisearch --http-addr 127.0.0.1:7700 --env production --master-key *****
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Security configuration

Security is an important aspect when configuring Meilisearch. We have created two keys to meet the security requirements. The master key, which allows full access to the Meilisearch instance, is used exclusively in the backend. The API key, on the other hand, which has more limited permissions, is used for search operations and other user tasks. This separation ensures that critical backend operations are securely protected while interactions in the frontend remain efficient and secure.

Integration with Laravel

Our platform TutKit.com is built on the PHP framework Laravel, and we have used the MeiliSearch-PHP library as the basis for the integration. The API key is used in our Laravel application to connect to Meilisearch. This significantly increases security, as only the functions allowed by the API key are accessible, minimizing the risk of unauthorized actions.

// Beispiel für die Konfiguration von Meilisearch in Laravel
'connections' => [
    'meilisearch' => [
        'host' => env('MEILISEARCH_HOST', 'http://localhost:7700'),
        'key' => env('MEILISEARCH_KEY', null),
    ],
],

Indexing and Data Management

A central aspect of our implementation was the efficient management and indexing of data. We developed a custom console command in Laravel that is specifically designed for extensive updates or for initiating the indexing process. This command specifically targets only the relevant records that need to be indexed. This allows us to avoid unnecessary data processing and focus on the essential information.

// Beispiel für einen benutzerdefinierten Konsolenbefehl in Laravel
Artisan::command('index:content', function () {
    // Code zur Indexierung der relevanten Datensätze
})->describe('Indexiere alle relevanten Inhalte');

The example is, of course, very simple. Our command code has 400 lines.

Automatic update of indices in the search.

A key advantage of our implementation is the ability to automatically detect and respond to changes in Laravel models. Whenever data is updated, added, or deleted, the corresponding MeiliSearch index is promptly and efficiently updated to reflect only those changes. This targeted indexing approach eliminates the need for a full reindexing of all data and ensures that our MeiliSearch data remains up to date. Here is a code example when a blog post has been updated:

if($this->status == 'PUBLIC') {
                $post_filtered = $this->prepareSearchField();
                $searchClient->index('suggestions')->updateDocuments([
                    [
                        'id' => 'blog_post_lang_'.$this->id,
                        'language' => $this->language->code,
                        'suggestion' => SearchHelper::prepareSearchSuggestion($this->title),
                        'indexation_type' => 'suggestion',
                        'type' => 'blog_post',
                    ]
                ]);

                $searchClient->index('blog_posts')->updateDocuments([$post_filtered]);
            } else {
                $searchClient->index('blog_posts')->deleteDocument($this->id);
                $searchClient->index('suggestions')->deleteDocument('blog_post_lang_'.$this->id);
            }

This code example is also extremely simple, and our lines of code operate on different models and different statuses. But in principle, it is clear what it’s about.

Optimization and Maintenance

Regular maintenance and optimization of the Meilisearch integration is crucial for long-term performance. By utilizing the regular updates from Meilisearch, we benefit from continuous improvements and new features. This ensures that our search solution remains up-to-date and can keep pace with the growing demands of our platform.

If you want to consider Meilisearch for your project, your developers can find documentation on the Meilisearch implementation here.

Here in the small animation, it becomes clear:

• the search is ultra fast
• The search is capable of handling typos and still shows the correct results.
• the instant search results appear after just a few letters
• The search filters allow for a quick narrowing down of the search results.

The effects of Meiliseach at TutKit.com

The integration of Meilisearch at TutKit.com by our own development team has proven to be a great success. With a fast and effective search solution that considers multilingualism and large volumes of data, we have significantly improved the user experience. Our self-developed implementation allows us to tailor the search function precisely to our needs while maintaining the control and security necessary for a powerful content platform.

With Meilisearch, we have achieved a significant improvement in the speed and efficiency of our search function. The indexing of new data is now lightning-fast, and the server load has been significantly reduced. Our users benefit from fast and relevant search results, greatly enhancing the user experience and reducing frustration from long loading times.

Financially, we have saved significant costs by using the open-source version of Meilisearch. Additionally, the high performance of Meilisearch allows us to manage without external support, further reducing our expenses.

We are excited to see how Meilisearch will develop further. We hope that future versions will integrate even more AI features that capture user search intent better and reduce the need for filters and categorization options. Meilisearch has proven to be the right choice for us.

In conclusion: We now have the perfect search engine with Meilisearch that helps us provide a fast, multilingual, and user-friendly search solution for our website users… in our assessment, it is currently the optimal search technology for enterprise challenges with hundreds of thousands or millions of contents.