Elasticsearch
是一个分布式的RESTful风格的搜索和数据分析引擎,能够解决不断涌现出的各种用例。作为Elastic Stack的核心,它存储您的数据,帮助您发现意料之中以及意料之外的情况。
特点:
- 实时搜索
- 速度快
- 分布式(无序人工搭建集群)
- Rest风格
1.安装
虚拟机安装Elasticsearch
上传压缩包
解压
切换到leyou用户(elasticsearch不能使用root账户运行)
$ id leyou改变文件的权限为属于leyou
$ chown leyou:leyou elasticsearch-6.2.4. -R修改文件名称
$ mv elasticsearch-6.2.4 elasticsearch配置jvm
$ vim /home/leyou/elasticsearch/config/jvm.options修改jvm堆内存大小,
-Xms1g改为-Xms256m,-Xmx1g改为Xmx256m配置elasticsearch.yml
$ vim /home/leyou/elasticsearch/config/elasticsearch.yml修改数据目录为:
path.data: /home/leyou/elasticsearch/data修改日志目录为:
path.logs: /home/leyou/elasticsearch/logs修改主机地址为:
network.host: 0.0.0.0创建data目录
$ mkdir /home/leyou/elasticsearch/data将data目录设置所属为leyou
$chown leyou:leyou . -R使用leyou用户启动
$ /home/leyou/elasticsearch/bin/elasticsearch出现错误信息:
ERROR: [4] bootstrap checks failed [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535] [2]: max number of threads [3069] for user [leyou] is too low, increase to at least [4096] [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [4]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured ERROR: Elasticsearch did not exit normally - check the logs at /home/leyou/elasticsearch/logs/elasticsearch.log10.1.用root账号修改配置文件
$vim /etc/security/limits.conf添加以下内容:
* soft nofile 65536 * hard nofile 131072 * soft nproc 4096 * hard nproc 409610.2.修改进程数量
$ vim /etc/security/limits.d/90-nproc.conf添加
* soft nproc 409610.3.修改虚拟内存
$ vim /etc/sysctl.conf添加
vm.max_map_count=655360执行
$ sysctl -p出现
vm.max_map_count = 655360表示配置生效10.4.重新连接虚拟机
10.5.启动
$ /home/leyou/elasticsearch/bin/elasticsearch
本机安装Kibana
下载安装包
解压
配置
修改config下的kibana.yml
elasticsearch.hosts: ["http://192.168.116.128:9200"]双击kibana.bat运行
测试访问
localhost:5601dev-tools
POST _analyze { "analyzer":"ik_max_word", "text":"我是中国人" }结果
{ "tokens" : [ { "token" : "我", "start_offset" : 0, "end_offset" : 1, "type" : "CN_CHAR", "position" : 0 }, { "token" : "是", "start_offset" : 1, "end_offset" : 2, "type" : "CN_CHAR", "position" : 1 }, { "token" : "中国人", "start_offset" : 2, "end_offset" : 5, "type" : "CN_WORD", "position" : 2 }, { "token" : "中国", "start_offset" : 2, "end_offset" : 4, "type" : "CN_WORD", "position" : 3 }, { "token" : "国人", "start_offset" : 3, "end_offset" : 5, "type" : "CN_WORD", "position" : 4 } ] }POST _analyze { "analyzer":"ik_smart", "text":"我是中国人" }结果
{ "tokens" : [ { "token" : "我", "start_offset" : 0, "end_offset" : 1, "type" : "CN_CHAR", "position" : 0 }, { "token" : "是", "start_offset" : 1, "end_offset" : 2, "type" : "CN_CHAR", "position" : 1 }, { "token" : "中国人", "start_offset" : 2, "end_offset" : 5, "type" : "CN_WORD", "position" : 2 } ] }
安装Elasticsearch-analysis-ik分词器
- 下载安装包
- 上传到
/home/leyou/elasticsearch/plugins/ - 解压
- 重新启动elasticsearch
2.ElasticSearch API 和操作索引
Rest风格的API,即http请求接口
2.1 基本概念
对比关系
索引集(indexs)–Databases数据库
类型(type)—Table数据库表
文档(Document)—Row行
字段(Field)—Columns列
一个索引库里面只有一种类型的文档,对应数据库中一种类型的数据
索引集(indexs):逻辑上的完整索引
分片(shard):数据拆分后的各个部分
副本(replica):每个分片的复制
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vYLREkLL-1588653806018)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200430173820012.png)]
2.2 创建索引
Elasticsearch采用Rest风格API,因此其API就是一次http请求,你可以用任何工具发起http请求
创建索引的请求格式
- 请求方式:PUT
- 请求路径:/索引名称
- 请求参数:json格式
{
"settings":{
"number_of_shards":3,
"number_of_replicas":2
}
}
返回结果
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "heima"
}
表示创建索引成功
使用kibana
PUT /hema2
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
}
}
结果
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "hema2"
}
使用kibana查询hema2
GET /hema2
结果
{
"hema2" : {
"aliases" : { },
"mappings" : { },
"settings" : {
"index" : {
"creation_date" : "1588240621231",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "njLBk0vGSOe01pAxNeEYBQ",
"version" : {
"created" : "7060299"
},
"provided_name" : "hema2"
}
}
}
}
删除hema2
DELETE /hema2
查询hema2
GET /hema2
结果显示404
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [hema2]",
"resource.type" : "index_or_alias",
"resource.id" : "hema2",
"index_uuid" : "_na_",
"index" : "hema2"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [hema2]",
"resource.type" : "index_or_alias",
"resource.id" : "hema2",
"index_uuid" : "_na_",
"index" : "hema2"
},
"status" : 404
}
2.3 创建映射字段
请求方式:PUT
PUT /索引库名/_mapping/类型名称
{
"properties":{
"字段名":{
"type":"类型",
"index":true,
"store":true,
"analyzer":"分词器"
}
}
}
- 类型名称:前面type的概念
- type:类型,可以实text、long、short、date、integer、object等
- index:是否索引,默认为true
- store:是否存储,默认为false
- analyzer:分词器,这里的
ik_max_word即使用ik分词器
示例
PUT /heima/_mapping
{
"properties":{
"title":{
"type":"text",
"analyzer":"ik_max_word"
},
"images":{
"type":"keyword",
"index":"false"
},
"price":{
"type":"float"
}
}
}
"type":"text",text类型是可分词的、"index":"false"表示不用于索引
返回结果
{
"acknowledged" : true
}
查询
GET /heima/
结果
{
"heima" : {
"aliases" : { },
"mappings" : {
"properties" : {
"images" : {
"type" : "keyword",
"index" : false
},
"price" : {
"type" : "float"
},
"title" : {
"type" : "text",
"analyzer" : "ik_max_word"
}
}
},
"settings" : {
"index" : {
"creation_date" : "1588240438655",
"number_of_shards" : "3",
"number_of_replicas" : "2",
"uuid" : "vikINtFSQ7KFwThXTeF9WA",
"version" : {
"created" : "7060299"
},
"provided_name" : "heima"
}
}
}
}
字段可加的属性:
type,表示字段的类型,常用的又
string、numerical、date还有复杂类型如对象,例如{gril:{name:“rose”,age:21}}会被处理成两个字段,
gril.name和gril.ageindex,
true表示能根据这个字段搜索,false表示不能根据这个字段搜索、默认值为truestore,默认值是
false,在elasticsearch中不通过该字段来设置一个字段要不要显示,即便是false也能搜索到结果,原因是elseticsearch在创建文档索引的时候,会将文档中的原始数据备份,保存到一个_source的属性中,而且我们可以通过过滤_source来决定哪些要显示,哪些不要显示
2.4 新增数据
demo1 通过POST请求,向索引库中添加数据
POST /索引库名/类型名
{
"key":"value"
}
示例
POST /heima/_doc
{
"title":"小米手机",
"images":"http://image.leyou.com/123.jpg",
"price":2699.0
}
结果
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
demo2 每条数据都有一个_id,上述id为自动生成,其值也能自己指定,如下
POST /heima/_doc/1
{
"title":"大米手机",
"images":"http://image.leyou.com/123.jpg",
"price":2899.0
}
结果
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1
}
demo3 查询所有
GET /heima/_search
{
"query": {"match_all": {}}
}
结果
{
"took" : 915,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 1.0,
"_source" : {
"title" : "小米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2699.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
}
]
}
}
demo4 查询
GET /heima/_doc/1
结果
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"_seq_no" : 1,
"_primary_term" : 1,
"found" : true,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
}
demo5 新增数据,该数据字段数目与之前的不同
POST /heima/_doc/3
{
"title":"超米手机",
"images":"http://image.leyou.com",
"price":2399,
"stock":200,
"saleable":true,
"subTitle":"呵呵"
}
响应:
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 4
}
查询所有
GET /heima/_search
{
"query": {"match_all": {}}
}
响应:
{
"took" : 134,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 4,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"title" : "超米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 1.0,
"_source" : {
"title" : "小米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2699.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "lcv-zXEB8nnTwNGWdjnf",
"_score" : 1.0,
"_source" : {
"title" : "超米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
}
]
}
}
查询映射
GET /heima/_mapping
发现多了两个字段
{
"heima" : {
"mappings" : {
"properties" : {
"images" : {
"type" : "keyword",
"index" : false
},
"price" : {
"type" : "float"
},
"saleable" : {
"type" : "boolean"
},
"stock" : {
"type" : "long"
},
"subTitle" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"title" : {
"type" : "text",
"analyzer" : "ik_max_word"
}
}
}
}
}
stock和saleable被成功映射了
如果存储的是string类型数据,elasticsearch会存入两个字段
- name:text类型
- name.keyword:keyword类型
2.5 修改数据
PUT请求方式,请求参数和POST新增数据时一样,但是请求要指定数据的id,如果id存在,则表示更新数据
如果id不存在的情况下,则为新增数据
demo1 id存在的情况下,表示更新数据
PUT /heima/_doc/3
{
"title" : "超大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
响应:
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_version" : 2,
"result" : "updated",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 4
}
result的值为update,表示更新
demo2 id不存在的情况下,表示新增数据:
PUT /heima/_doc/2
{
"title" : "不存在的大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
响应:
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 3,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 3,
"_primary_term" : 4
}
result的值为created,表示创建了数据
2.6 删除数据
根据id进行删除
DELETE /索引库名/类型名/id值
3.查询
3.1 全文检索查询
查询模板
GET /索引库名/_search
{
"query": {"match": {
"FIELD": "TEXT"
}}
}
需要指定字段名FIELD和要查询的内容TEXT
demo1 查询title字段为“大米手机”的数据
GET /heima/_search
{
"query": {
"match": {
"title": "大米手机"
}
}
}
响应:
{
"took" : 64,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.1835277,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.1835277,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.42221838,
"_source" : {
"title" : "超大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.320886,
"_source" : {
"title" : "不存在的大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 0.14181954,
"_source" : {
"title" : "小米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2699.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "lcv-zXEB8nnTwNGWdjnf",
"_score" : 0.11955717,
"_source" : {
"title" : "超米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
}
]
}
}
因为查询条件,和索引库中的数据都做了分词,所以与“大米手机”有关的数据都查询出来了,这种查询,词条之间的关系为or
在查询时词条之间采用and的方式,则使用如下的方式进行查询
GET /heima/_search
{
"query": {
"match": {
"title": {
"query":"大米手机",
"operator": "and"
}
}
}
}
响应:
{
"took" : 18,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.1835277,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.1835277,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.42221838,
"_source" : {
"title" : "超大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.320886,
"_source" : {
"title" : "不存在的大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
}
]
}
}
查询所有
GET /heima/_search
{
"query": {
"match_all": {}
}
}
- query代表查询对象
- match_all代表查询所有
响应:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"title" : "不存在的大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"title" : "超大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 1.0,
"_source" : {
"title" : "小米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2699.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "lcv-zXEB8nnTwNGWdjnf",
"_score" : 1.0,
"_source" : {
"title" : "超米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
}
}
]
}
}
- took:花费时间,单位是毫秒
- time_out:是否超时
- _shards:分片信息
- hits:搜索结果总览对象
- total:搜索到的总条数
- max_score:结果文档中的最高得分
- hits:搜索到的全部文档的对象数组
- _index:索引库
- _type:文档类型
- _id:文档id
- _score:文档得分
- _source:文档的源数据
or和and之间
match支持minimum_should_match最小匹配数查询
GET /heima/_search
{
"query": {
"match": {
"title": {
"query":"大米手机",
"minimum_should_match": "75%"
}
}
}
}
minimum_should_match设置为一个百分数,表示匹配到总词条数的75%即可
3.2 多字段搜索
GET /heima/_search
{
"query": {
"multi_match": {
"query": "小米",
"fields": ["title","subTitle"]
}
}
}
3.3 词条搜索
GET /heima/_search
{
"query": {
"term": {
"title": {
"value": "大米手机"
}
}
}
}
将要查询的内容视为一个词条,不再对查询内容进行分词
查询结果
{
"took" : 8,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
因此查询结果为0个,因为没有这个词条
词条查询适用于不分词的字段,除了text类型的字段外,其他类型的字段均不能分词
3.4 多词条精确匹配
GET /heima/_search
{
"query": {
"terms": {
"price":[2399.00,2899.00]
}
}
}
3.5 结果过滤
设置_source:"title",表示结果中只要求title字段
GET /heima/_search
{
"_source": "title",
"query": {
"match": {
"title":"大米手机"
}
}
}
响应:
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.1835277,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.1835277,
"_source" : {
"title" : "大米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.42221838,
"_source" : {
"title" : "超大米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.320886,
"_source" : {
"title" : "不存在的大米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 0.14181954,
"_source" : {
"title" : "小米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "lcv-zXEB8nnTwNGWdjnf",
"_score" : 0.11955717,
"_source" : {
"title" : "超米手机"
}
}
]
}
}
可以看到_source中只包含title字段,这样做能提高搜索效率
如果保留多个字段
GET /heima/_search
{
"_source": ["title","price"],
"query": {
"match": {
"title":"大米手机"
}
}
}
如果排除多个字段
GET /heima/_search
{
"_source": {
"excludes": "images"
},
"query": {
"match": {
"title":"大米手机"
}
}
}
查询结果
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.1835277,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.1835277,
"_source" : {
"price" : 2899.0,
"title" : "大米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : 0.42221838,
"_source" : {
"subTitle" : "呵呵",
"price" : 2399,
"saleable" : true,
"title" : "超大米手机",
"stock" : 200
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : 0.320886,
"_source" : {
"subTitle" : "呵呵",
"price" : 2399,
"saleable" : true,
"title" : "不存在的大米手机",
"stock" : 200
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4H3wynEBcst0HpnrTPRf",
"_score" : 0.14181954,
"_source" : {
"price" : 2699.0,
"title" : "小米手机"
}
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "lcv-zXEB8nnTwNGWdjnf",
"_score" : 0.11955717,
"_source" : {
"subTitle" : "呵呵",
"price" : 2399,
"saleable" : true,
"title" : "超米手机",
"stock" : 200
}
}
]
}
}
3.6 高级搜索
模糊查询
POST /heima/_doc/4
{
"title":"apple",
"images":"http://image.leyou.com",
"price":6899.00
}
GET /heima/_search
{
"query": {
"fuzzy": {
"title": "appla"
}
}
}
{
"took" : 246,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0995283,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "4",
"_score" : 1.0995283,
"_source" : {
"title" : "apple",
"images" : "http://image.leyou.com",
"price" : 6899.0
}
}
]
}
}
范围查询
查询价格大于1000,小于3000的数据
GET /heima/_search
{
"query": {
"range": {
"price": {
"gte": 1000,
"lte": 3000
}
}
}
}
布尔查询
GET /heima/_search
{
"query": {
"bool": {
"must": [
{"match": {
"title": "apple"
}},
{
"range": {
"price": {
"gte": 3000
}
}
}
]
}
}
}
查询title既能匹配apple并且price大于3000的数据
must:条件必须同时成立should:至少有一个条件成立must_not:条件必须同时都不成立
must的条件会影响的分,有range条件和没有range条件时,查询结果的得分不同
GET /heima/_search
{
"query": {
"bool": {
"must": [
{"match": {
"title": "大米"
}},
{
"range": {
"price": {
"gte": 1000
}
}
}
]
}
}
}
如果不想must条件影响得分可以使用filter
GET /heima/_search
{
"query": {
"bool": {
"must": [
{"match": {
"title": "大米"
}}
],
"filter": [
{"range": {
"price": {
"gte": 1000
}
}}
]
}
}
排序
根据price字段倒序排序
GET /heima/_search
{
"query": {
"bool": {
"must": [
{"match": {
"title": "大米"
}}
],
"filter": [
{"range": {
"price": {
"gte": 1000
}
}}
]
}
},
"sort": [
{
"price": {
"order": "desc"
}
}
]
}
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "1",
"_score" : null,
"_source" : {
"title" : "大米手机",
"images" : "http://image.leyou.com/123.jpg",
"price" : 2899.0
},
"sort" : [
2899.0
]
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "2",
"_score" : null,
"_source" : {
"title" : "不存在的大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
},
"sort" : [
2399.0
]
},
{
"_index" : "heima",
"_type" : "_doc",
"_id" : "3",
"_score" : null,
"_source" : {
"title" : "超大米手机",
"images" : "http://image.leyou.com",
"price" : 2399,
"stock" : 200,
"saleable" : true,
"subTitle" : "呵呵"
},
"sort" : [
2399.0
]
}
]
}
}
分页
GET /heima/_search
{
"query": {
"bool": {
"must": [
{"match": {
"title": "大米"
}}
],
"filter": [
{"range": {
"price": {
"gte": 1000
}
}}
]
}
},
"sort": [
{
"price": {
"order": "desc"
}
}
],
"from": 0,
"size": 2
}
from:起始页size:页大小
4.聚合
聚合常用的两种类型:桶和度量
聚合为桶的划分依据:
根据词条内容
根据数值阶梯划分
根据日期阶梯划分
根据数值和日期的范围
常用的度量聚合方式:
- 求平均值
- 求最大最小值
- 求百分比
- 求和
- 求前几
- 求总数
- 求avg、max、min、sum、count
示例
添加索引
PUT /cars
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"color":{
"type":"keyword"
},
"make":{
"type": "keyword"
}
}
}
}
添加数据
POST /cars/_doc/_bulk
{"index":{}}
{"price":10000,"color":"red","make":"honda","sold":"2014-10-28"}
{"index":{}}
{"price":20000,"color":"red","make":"honda","sold":"2014-11-05"}
{"index":{}}
{"price":30000,"color":"green","make":"ford","sold":"2014-05-18"}
{"index":{}}
{"price":15000,"color":"blue","make":"toyota","sold":"2014-07-12"}
{"index":{}}
{"price":12000,"color":"green","make":"toyota","sold":"2014-08-19"}
{"index":{}}
{"price":20000,"color":"red","make":"honda","sold":"2014-11-05"}
{"index":{}}
{"price":80000,"color":"red","make":"bmw","sold":"2014-01-01"}
{"index":{}}
{"price":25000,"color":"blue","make":"ford","sold":"2014-02-12"}
根据制造商查询
GET /cars/_search
{
"size": 0,
"aggs": {
"max_popular": {
"terms": {
"field": "make"
}
}
}
}
查询结果
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 8,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"max_popular" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "honda",
"doc_count" : 3
},
{
"key" : "ford",
"doc_count" : 2
},
{
"key" : "toyota",
"doc_count" : 2
},
{
"key" : "bmw",
"doc_count" : 1
}
]
}
}
}
size表示搜索结果,这是将搜索结果设置为0,aggregations表示聚合结果,max_popular表示聚合的名字,buckets表示桶,key表示分桶的字段,doc_count表示这一个桶有多少条数据。
求平均值
GET /cars/_search
{
"size": 0,
"aggs": {
"popular_brand": {
"terms": {
"field": "make"
},
"aggs": {
"price_avg": {
"avg": {
"field": "price"
}
}
}
}
}
}
结果
{
"took" : 229,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 8,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"popular_brand" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "honda",
"doc_count" : 3,
"price_avg" : {
"value" : 16666.666666666668
}
},
{
"key" : "ford",
"doc_count" : 2,
"price_avg" : {
"value" : 27500.0
}
},
{
"key" : "toyota",
"doc_count" : 2,
"price_avg" : {
"value" : 13500.0
}
},
{
"key" : "bmw",
"doc_count" : 1,
"price_avg" : {
"value" : 80000.0
}
}
]
}
}
}
创建统计图
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ES0CiBhL-1588653806025)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200502105054362.png)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xHmUH9Ll-1588653806028)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200502105715975.png)]
动态仪表盘
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-AVzaEMtj-1588653806033)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200502110044320.png)]
5.Elasticsearch java客户端
创建一个maven项目 groupId:com.leyou.demo,ArtifactId:es-demo
引入依赖
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.leyou.demo</groupId> <artifactId>es-demo</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <name>elasticsearch</name> <description>Demo project for Spring Boot</description> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.2.5.RELEASE</version> <relativePath/> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>application.yaml
spring: data: elasticsearch: cluster-name: elasticsearch cluster-nodes: 192.168.116.128:9300创建启动类
package com.leyou; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class EsApplication { public static void main(String[] args) { SpringApplication.run(EsApplication.class); } }
5.1 创建索引库
创建实体类Item
package com.leyou.es.pojo; import lombok.Data; import org.springframework.data.annotation.Id; import org.springframework.data.elasticsearch.annotations.Document; import org.springframework.data.elasticsearch.annotations.Field; import org.springframework.data.elasticsearch.annotations.FieldType; @Data @Document(indexName = "heima3",type = "item",shards = 1) public class Item { @Field(type = FieldType.Long) @Id Long id; @Field(type = FieldType.Text,analyzer = "ik_smart") String title; @Field(type = FieldType.Keyword) String category; @Field(type = FieldType.Keyword) String brand; @Field(type = FieldType.Double) Double price; @Field(type = FieldType.Keyword,index = false) String images; }创建测试类
package com.leyou.es.demo; import com.leyou.es.pojo.Item; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.data.elasticsearch.core.ElasticsearchTemplate; import org.springframework.test.context.junit4.SpringRunner; @RunWith(SpringRunner.class) @SpringBootTest public class EsTest { @Autowired ElasticsearchTemplate template; @Test public void testCreate(){ // 创建索引库 template.createIndex(Item.class); // 创建映射 template.putMapping(Item.class); } }执行测试类,添加索引库和映射关系
查询
GET /heima3/_mapping结果
{ "heima3" : { "mappings" : { "properties" : { "brand" : { "type" : "keyword" }, "category" : { "type" : "keyword" }, "id" : { "type" : "keyword" }, "images" : { "type" : "keyword", "index" : false }, "price" : { "type" : "double" }, "title" : { "type" : "text", "analyzer" : "ik_smart" } } } } }
5.2 批量新增数据
新建一个ItemRepository接口继承ElasticsearchRepository
package com.leyou.es.repository; java import com.leyou.es.pojo.Item; import org.springframework.data.elasticsearch.repository.ElasticsearchRepository; public interface ItemRepository extends ElasticsearchRepository<Item,Long> { }批量新增数据
@Test public void insetIndex(){ List<Item> items = new ArrayList<>(); items.add(new Item(1L,"小米手机7","手机","小米",3299.00,"http://image.leyou.com/123.jpg")); items.add(new Item(2L,"坚果手机R1","手机","锤子",3699.00,"http://image.leyou.com/123.jpg")); items.add(new Item(3L,"华为手机MATE10","手机","华为",4499.00,"http://image.leyou.com/123.jpg")); items.add(new Item(4L,"小米手机Min2S","手机","小米",4299.00,"http://image.leyou.com/123.jpg")); items.add(new Item(5L,"荣耀V10","手机","华为",2799.00,"http://image.leyou.com/123.jpg")); itemRepository.saveAll(items); }查询
GET /heima3/_search { "query": {"match_all": {}} }结果
{ "took" : 158, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 5, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "heima3", "_type" : "item", "_id" : "5", "_score" : 1.0, "_source" : { "id" : 5, "title" : "荣耀V10", "category" : "手机", "brand" : "华为", "price" : 2799.0, "images" : "http://image.leyou.com/123.jpg" } }, { "_index" : "heima3", "_type" : "item", "_id" : "2", "_score" : 1.0, "_source" : { "id" : 2, "title" : "坚果手机R1", "category" : "手机", "brand" : "锤子", "price" : 3699.0, "images" : "http://image.leyou.com/123.jpg" } }, { "_index" : "heima3", "_type" : "item", "_id" : "3", "_score" : 1.0, "_source" : { "id" : 3, "title" : "华为手机MATE10", "category" : "手机", "brand" : "华为", "price" : 4499.0, "images" : "http://image.leyou.com/123.jpg" } }, { "_index" : "heima3", "_type" : "item", "_id" : "4", "_score" : 1.0, "_source" : { "id" : 4, "title" : "小米手机Min2S", "category" : "手机", "brand" : "小米", "price" : 4299.0, "images" : "http://image.leyou.com/123.jpg" } }, { "_index" : "heima3", "_type" : "item", "_id" : "1", "_score" : 1.0, "_source" : { "id" : 1, "title" : "小米手机7", "category" : "手机", "brand" : "小米", "price" : 3299.0, "images" : "http://image.leyou.com/123.jpg" } } ] } }
5.3 查询
@Test
public void testFind(){
Iterable<Item> all = itemRepository.findAll();
for (Item item : all) {
System.out.println(item);
}
}
结果
Item(id=5, title=荣耀V10, category=手机, brand=华为, price=2799.0, images=http://image.leyou.com/123.jpg)
Item(id=2, title=坚果手机R1, category=手机, brand=锤子, price=3699.0, images=http://image.leyou.com/123.jpg)
Item(id=3, title=华为手机MATE10, category=手机, brand=华为, price=4499.0, images=http://image.leyou.com/123.jpg)
Item(id=4, title=小米手机Min2S, category=手机, brand=小米, price=4299.0, images=http://image.leyou.com/123.jpg)
Item(id=1, title=小米手机7, category=手机, brand=小米, price=3299.0, images=http://image.leyou.com/123.jpg)
5.4 复杂查询,根据价格范围查询(自定义查询)
在接口中写一个根据价格范围查询的方法
package com.leyou.es.repository; import com.leyou.es.pojo.Item; import org.springframework.data.elasticsearch.repository.ElasticsearchRepository; import java.util.List; public interface ItemRepository extends ElasticsearchRepository<Item,Long> { public List<Item> findByPriceBetween(Double start, Double end); }测试范围查询
@Autowired ItemRepository itemRepository; @Test public void testFindBy(){ List<Item> itemList = itemRepository.findByPriceBetween(2000d, 4000d); for (Item item : itemList) { System.out.println(item); } }结果
Item(id=5, title=荣耀V10, category=手机, brand=华为, price=2799.0, images=http://image.leyou.com/123.jpg) Item(id=2, title=坚果手机R1, category=手机, brand=锤子, price=3699.0, images=http://image.leyou.com/123.jpg) Item(id=1, title=小米手机7, category=手机, brand=小米, price=3299.0, images=http://image.leyou.com/123.jpg)
Spring Data Elasticsearch 这种自定义查询不能使用聚合,聚合要使用elasticsearch的原生API
5.5 使用elasticsearch的原生API查询
@Autowired
ItemRepository itemRepository;
@Test
public void testQuery(){
// 创建查询构建器
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
// 结果过滤
queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{"id", "title", "price"}, null));
// 添加查询条件
queryBuilder.withQuery(QueryBuilders.matchQuery("title", "小米手机"));
// 倒序排序
queryBuilder.withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC));
// 分页
queryBuilder.withPageable(PageRequest.of(0, 2));
Page<Item> result = itemRepository.search(queryBuilder.build());
long totalElements = result.getTotalElements();
System.out.println("totalElements=" + totalElements);
int totalPages = result.getTotalPages();
System.out.println("totalPages=" + totalPages);
List<Item> content = result.getContent();
for (Item item : content) {
System.out.println("item="+item);
}
}
查询结果
totalElements=4
totalPages=2
item=Item(id=3, title=华为手机MATE10, category=null, brand=null, price=4499.0, images=null)
item=Item(id=4, title=小米手机Min2S, category=null, brand=null, price=4299.0, images=null)
查询构建器时Spring Data提供的,通过构建器添加查询和过滤条件,这些查询和过滤条件时elasticsearch提供的,然后使用Spring Data 提供的itemRepository进行搜索,其返回值是一个分页结果,上面查询的结果和kibana查询结果相同
GET /heima3/_search
{
"query": {
"match": {
"title": "小米手机"
}
},
"_source": ["id","title","price"],
"sort": [
{
"price": {
"order": "desc"
}
}
],
"from": 0,
"size": 2
}
kibana中的from是指数据起始位置,而程序中分页时用的page,指的是起始页
5.6 聚合查询,根据品牌聚合
@Autowired
ElasticsearchTemplate template;
@Test
public void testAggs(){
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
String aggName = "popular_brand";
// 聚合
queryBuilder.addAggregation(AggregationBuilders.terms(aggName).field("brand"));
// 查询并返回聚合结果
AggregatedPage<Item> result = template.queryForPage(queryBuilder.build(), Item.class);
// 解析聚合结果
Aggregations aggs = result.getAggregations();
// 指定聚合名称的聚合
StringTerms terms = aggs.get(aggName);
// 获取桶
List<StringTerms.Bucket> buckets = terms.getBuckets();
for (StringTerms.Bucket bucket : buckets) {
System.out.println("key=" + bucket.getKey());
System.out.println("docCount=" + bucket.getDocCount());
}
}
查询结果
key=华为
docCount=2
key=小米
docCount=2
key=锤子
docCount=1
等价的kibana查询
GET /heima3/_search
{
"size": 0,
"aggs": {
"popular_brand": {
"terms": {
"field": "brand"
}
}
}
}