Selecting a Search Engine for on-site Search — Open Source vs Search as a Service

On-site search (also known as site search or internal search) allows users to identify the required data from a website by typing queries into a search bar.

The website search helps the visitors to quickly navigate and identify the required content within the website, the website search reduces bounce rate, boosts conversions also helps us to identify what visitors are more interested/looking for in our website. The website search is now an essential part of website design, when your website search doesn’t function as expected it will impact the website visitors(potentially lose the customers).

Photo by Markus Winkler on Unsplash

The important factor is to identify the best search engine/platform that helps you to enable the required website search functionalities. The search platform should be identified not only considering the current business requirements but the future needs — in some cases machine learning capabilities might not be a requirement today but it will become the mandatory requirement in the future.

There are two major categories of search engines:

• Build an internal search engine from scratch
• Open Source
• Search as a Service

Open source search is implemented through free, open-source software that is built to handle many different use cases. Typically, the software is governed by the Apache or MIT License, which effectively allows for any commercial or personal use. The internal team should take care of the hosting, operations, maintenance(external hosting options are possible)

In Search as a Service, search functionality, hosting, operations, maintenance, and more are provided by the software provider.

Features:

Some of the important features to be considered while selecting the search platforms:

• Auto-Complete
• Multi-language support
• Boost or Bury content — the search admins should be able to boost or burry the content quickly based on the need through simplified UI
• Search Performance — the search result should be displayed based on the business requirement e.g <2s
• Ranking/Relevance — Easy to enable different ranking and relevance rules, should be able to be easily managed through admin UI’s
• Hosting options — Cloud/SaaS solution
• Seamless Integration with different platforms(Index Connectors) — Easy integration for indexing the data from various sources
• Faceted Search
• Federated Search — Search across various sources and display the consolidated view to the site visitors.
• Analytic Capabilities
• Easy Search administration — easy search administration UI’s
• Access management — Easy access management to control search admin access based on the roles
• Machine Learning capabilities — Machine learning models to increase the search relevance.
• Search within documents
• Natural Language Search — Support Natural Language Search, natural language search is a search using regular spoken language, such as English, the default search behavior is based on different keywords.
• Recommendations/Personalization capabilities — Capability to provide personalized/recommended search results based on visitor profile and search behaviors.
• Data Enrichment: Capability to enrich the crawled data to build a better index to provide relevant data to the visitors. In some cases various metadata may need to be derived from pages due to the lack of existing metadata in the pages — manually enabling the required metadata across all the pages will be cumbersome.
• Extract Metadata: In some cases, various metadata e.g page category, may need to be derived from the pages due to the lack of existing metadata in the pages — manually enabling the required metadata across all the pages will be cumbersome. The search engines should support the capability to discover different metadata from page content, URL, etc.
• Experience to different channels: Flexible options to deliver the search experience to different channels e.g API’s, SDK’s, UI frameworks to build the search experience.

The Open Source search engines e.g Apache Solr, Elastic Search, etc supports most of the basic search functionalities full-text search, Faceted search, Filtering, Auto-suggest and Spell check, etc. It requires a great level of customization/maintenance and development effort to make it suitable for your requirements. On other hand Search as Service provides basic and advanced search functionalities, hosting, operations, maintenance, support, etc

Considerations while selecting search platform:

Let us now see some of the considerations while deciding on the Search platform. Please note this my personal view based on my experience and use cases, will vary based on your need and requirements.

Engine vs Platform: Open source search engines e.g Apache Solr, Elastic Search, etc require a great level of development effort to make it a fully functional platform to meet all the requirements. On other hand Search as Service vendors provide the platform that provides the basic and advanced search functionalities, hosting, maintenance, support, etc.

Buy vs Build: Open source search engines set up and customization (Build) might take more time. On other hand Search as Service provides a prebuilt platform that can be used with minimal configuration.

SaaS solution: Most of the open-source search engines are not SaaS solutions (the hosting providers can be used in some cases), the maintainability effort is more for non-Saas-based solutions — platform maintenance/operation support.

Dependency on the Dev team: Maintenance, Scalability, Customization, Availability, and Security required more effort from the development team for open-source search engines. The new site launches required more effort and time from the development team to enable the search for the added websites. On the other hand, Search as Service by default takes care of Maintenance, Scalability, Availability, and Security. Search as Service enables easy management UI’s to enable the customizations, search configurations and, etc.

UI Capabilities for Search Management: In most cases, open-source search engines provide minimal UI capabilities to manage the search configurations, indexing, search rules, etc. A considerable amount of effort is required to build UI wrappers to support our requirements if the wrappers are not built the Dev team support is required to manage every configuration change, the marketing team will not be able to use the default UI’s to manage the search rules to boost or bury the search results — the custom UI wrappers need to be built if the requirement is the marketing team to manage the rules, if not Dev team should involved for every rule changes.

Support for advanced features: Some of the advanced features such as Machine Learning, Natural Language Search, Recommendations, Personalization, Federated Search, UI capabilities, etc will not be supported by default by open-source search engines, but it is possible to enable by customizing or enabling additional external toolsets — may require a considerable amount of effort. On other hand, most of the Search as a Service platforms support the advanced search features by default.

Analytics: Analytics is crucial for driving improvements and tweaks to the search tool, but they aren’t automatically included with open-source software — the Dev team should enable the required customizations to send the search data to analytic engines. On the other hand, most of the Search as a Service platform enables advanced analytic capabilities, also enables the capability to share analytic data with external analytics engines e.g GA 360, Adobe Analytics, etc.

Connectors for indexing: Indexing from different data sources e.g Web, REST services, eCommerce engines, Databases, Sharepoint, etc is one of the basic requirements while building the search platform, most of the Search as a Service platform enables connectors to easily fetch the data from variables sources for indexing. On the other hand, the open-source search engines required a considerable amount of dev effort/additional tools to index the data from various sources.

Support: Open source search tools are used and developed by software engineers all around the world. Online communities can be a great resource for your development teams. On the other hand, Search as Service vendors provide the support — in some cases, you may need to buy the support separately.

Cost: Open source search engines require minimal upfront costs but the total ownership cost may become more while enabling the complete platform to meet all the business requirements. On the other hand upfront cost will be huge for Search as a Service — the total ownership cost may become balanced in long run.

Technical Expertise: Open source search engines require a fair amount of technical expertise to build and maintain the platform. on the other hand, minimal team(some effort from the existing teams) should be enough to manage the Search as a Service platform.

Time to Launch sites with Search Capabilities: The time to launch the new websites with the search will be more for Open Source Search engines as the development team should be involved to configure the new with required search configurations(the effort will be reduced if a flexible framework is established to handle multiple sites). On the other hand, the effort will be less to launch the new site enabled with Search through Search as a Service, the admins with less technical knowledge can use the UI’s to enable the additional search configurations once after the initial platform configuration is enabled.

Flexibility/Control: The open-source search tools provide more flexibility for deployment, management, and customization. On the other hand loss of flexibility is inevitable for Search as a Service.

Conclusion:

Considering the timeline, required technical expertise, the effort for maintenance and support, advanced capabilities, ease of use to manage the search configuration/rules, Search as a Service makes more sense to quickly enable the platform and manage it with limited resources — there are multiple Search as a Service platform available in the market e.g Coveo, LucidWorks, Mindbreeze, etc

Open-source search engines support most of the basic functionalities required to support the website search but it requires a great level of customization/maintenance and development effort to meet all the required business requirements.

It should be great to connect with different agencies line Gartners and other vendors/partners to identify the best search tool that fits your current and future search needs.

In my perspective(will vary based on your understanding and requirements):

— If the upfront cost is not much concern(mainly licensing), also focus is on more advanced functionalities, ease of use for search management, timeline constraint it will make more sense to consider Search as a Service provider.

— If there is no constraint on timeline and resource(expertise) availability also not much focus on advanced capabilities, required complete control and flexibility, it will make more sense to consider Open Source Search engines.