Table of Contents- Inverted Index

• Elasticsearch Query and Storage Architecture
• Comparison with Database Queries:
• Basic Concepts
• Use Cases
• Core Components
• Working with Elasticsearch
• Index Operations
• Elasticsearch Data Types
• Mapping Operations
• Document Operations

Inverted Index

An inverted index is created by tokenizing documents to establish relationships between terms and their corresponding document IDs.

Using Tang Dynasty poetry as an example, finding verses containing the character "front"

Forward index: From "Quiet Night Thoughts" -> Moonlight before my window -> "front" character Inverted index: "front" character -> Moonlight before my window -> "Quiet Night Thoughts"

The implementation of an inverted index involves breaking down verses into individual words, then mapping from words to documents, which constitutes an inverted index.

key(term)	value	value
bed	moonlight before my bed	Quiet Night Thoughts
before	moonlight before my bed	Quiet Night Thoughts
before bed	moonlight before my bed	Quiet Night Thoughts
bright moon	moonlight before my bed	Quiet Night Thoughts
moon	moonlight before my bed	Quiet Night Thoughts

Elasticsearch Query and Storage Architecture

elasticsearch	relational database
index	database
mapping	table schema
document	table row

Comparison with Database Queries:

Database performance limitations: Using wildcard queries with leading wildcards prevents index utilization, resulting in full table scans which are inefficient.

Database functionality limitations: When searching for "Huawei mobile phones," databases typically don't perform tokenization, potentially missing relevant data.

term	value
Huawei	1
Telecom	1
3G	1
mobile	1, 2, 3

Basic Concepts

1. A search server built on Lucene
2. A distributed, highly scalable, real-time search and data analysis engine
3. Based on RESTful web interfaces
4. Elasticsearch is developed in Java and released under the Apache license as open source, making it a popular enterprise search solution
5. Official website: https://www.elastic.co/

Use Cases

1. Search: Querying massive datasets
2. Log data analysis
3. Real-time data analysis

Core Components

Concept	Description
Index	The location where Elasticsearch stores data, comparable to a database in relational systems.
Mapping	Mapping defines the data type of each field, the analyzer used, and other field-level configurations. Similar to table schema in relational databases.
Document	The smallest unit of data in Elasticsearch, typically represented in JSON format. A document corresponds to a row in a relational database.
Inverted Index	An inverted index consists of a list of all unique terms from documents, with each term pointing to a list of documents containing that term.
Type	A type represents a category of documents. In Elasticsearch 7.x, the default type is _doc.

Working with Elasticsearch

RESTful API Introduction

1. REST (Representational State Transfer)

Representational State Transfer is a set of architectural constraints and principles. Applications or designs that meet these constraints and principles are RESTful. It's essentially a specification for defining interfaces.

2. Based on HTTP.
3. Uses XML or JSON format for data representation.
4. Each URI represents a specific resource.
5. Clients use four HTTP verbs to manipulate server resources:

GET: Used to retrieve resources

POST: Used to create resources (can also be used for updates)

PUT: Used to update resources

DELETE: Used to delete resources

Index Operations

Create index

http://host:port/index_name

Query

GET http://host:port/index_name # Query single index information 
GET http://host:port/index_name1,index_name2... # Query multiple indices 
GET http://host:port/_all # Query all indices

Delete index

DELETE http://host:port/index_name

Close/open index

POST http://host:port/index_name/_close 
POST http://host:port/index_name/_open

Elasticsearch Data Types

1. String types

Aggregation: Similar to sum() in SQL

text: Tokenized, doesn't support aggregation 
keyword: Not tokenized, treats entire content as a single term, supports aggregation

2. Numeric types
3. Boolean: boolean
4. Binary: binary
5. Range types

integer_range, float_range, long_range, double_range, date_range

6. Date: date

Complex Data Types

Arrays: [ ] Nested: nested (for arrays of JSON objects)

Objects: { } Object: object (for single JSON objects)

Mapping Operations

PUT customer_index 

GET customer_index 
# Add mapping 
PUT /customer_index/_mapping 
{
	"properties":{
		"full_name":{ 
			"type":"text" 
		},
		"customer_age":{ 
			"type":"integer" 
		} 
	}
}

Create index with mapping

# Create index with mapping 
PUT /customer_profile { 
	"mappings": { 
		"properties": { 
			"full_name": {
				"type": "text" 
			},
			"customer_age": {
            	"type": "integer" 
            } 
		} 
	} 
}

GET customer_profile/_mapping

Add field

# Add field 
PUT /customer_profile/_mapping { 
	"properties": { 
		"full_name": { 
			"type": "text" 
		},
		"customer_age": { 
			"type": "integer" 
		} 
	}
}

Document Operations

Add document with specified ID

POST /customer_profile/_doc/2 
{ 
	"full_name":"John Smith", 
	"customer_age":28, 
	"location":"New York" 
}
	
GET /customer_profile/_doc/1

Add document without specified ID

# Add document without ID 
POST /customer_profile/_doc/ 
{ 
	"full_name":"Jane Doe", 
	"customer_age":32, 
	"location":"Los Angeles" 
}

# Query all documents GET /customer_profile/_search

Delete document by ID

# Delete document by ID 
DELETE /customer_profile/_doc/1