Docker部署gitlab、ELK
<p>[TOC]</p>
<h1>配置镜像加速器</h1>
<pre><code class="language-bash">sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://ifpcn5z8.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker</code></pre>
<h1>传统方式部署</h1>
<h2>1.搜索gitlab</h2>
<pre><code class="language-bash">docker search gitlab</code></pre>
<h2>2.下载gitlab</h2>
<pre><code class="language-bash">docker pull gitlab/gitlab-ce:latest
中文版：docker pull twang2218/gitlab-ce-zh:latest</code></pre>
<h2>3.查看镜像文件</h2>
<pre><code class="language-bash">docker images</code></pre>
<h2>4.创建gitlab挂载目录</h2>
<pre><code class="language-bash">mkdir -pv /data/docker/gitlab/{config,logs,data}</code></pre>
<h2>5.创建docker中的网络</h2>
<pre><code class="language-bash">docker network create gitlab_net</code></pre>
<h2>6.使用镜像启动Gitlab容器</h2>
<pre><code class="language-bash">docker run -d --net=gitlab_net -p 10443:443 -p 11110:80 -p 10022:22 --name gitlab -v /data/docker/gitlab/config/:/etc/gitlab -v /data/docker/gitlab/logs:/var/log/gitlab -v /data/docker/gitlab/data:/var/opt/gitlab gitlab/gitlab-ce:latest</code></pre>
<h2>7. 登录Gitlab,IP为宿主机IP地址 <a href="http://127.0.0.1:11110">http://127.0.0.1:11110</a> (如果访问上面的地址页面显示502，一般是端口冲突了，需要修改/data/docker/gitlab/config/gitlab.rb文件，重启docker容器即可.)</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=d9052ebf1bd58fd1f846c5f3457c4147&file=file.png" alt="" /></p>
<h1>docker-compose部署</h1>
<p>vim docker-compose.yml</p>
<pre><code class="language-yaml"></code></pre>
<h1>常用命令</h1>
<pre><code class="language-bash"># 启动所有 gitlab 组件；
gitlab-ctl start
# 停止所有 gitlab 组件；
gitlab-ctl stop
# 重启所有 gitlab 组件；
gitlab-ctl restart
# 查看服务状态；
gitlab-ctl status
# 修改gitlab配置文件；
vim /etc/gitlab/gitlab.rb
# 重新编译gitlab的配置；
gitlab-ctl reconfigure
# 检查gitlab；
gitlab-rake gitlab:check SANITIZE=true --trace
# 查看日志；
gitlab-ctl tail
gitlab-ctl tail nginx/gitlab_access.log
</code></pre>
<h1>部署ELK</h1>
<p><strong>Logstash：</strong>是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用。
<strong>Elasticsearch：</strong>是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
<strong>Kibana：</strong>是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。</p>
<p>三者关系：Logstash采集数据--->Elasticsearch存储数据--->Kibana可视化</p>
<p>下载地址：<a href="https://www.elastic.co/cn/downloads/">https://www.elastic.co/cn/downloads/</a></p>
<h2>Docker中国官方镜像加速</h2>
<pre><code>vim /etc/docker/daemon.json
#Docker中国官方镜像加速
{
"registry-mirrors": ["https://registry.docker-cn.com"]
}
#网易163镜像加速
{
"registry-mirrors": ["http://hub-mirror.c.163.com"]
}
#中科大镜像加速
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}</code></pre>
<h2>文件目录结构</h2>
<pre><code class="language-bash">[root@test ~]# tree elk/
elk/
├── data
│ ├── elasticsearch #用来保存elasticsearch数据用的
│ ├── logs #需要采集的日志的目录，也就是你应用写日志的位置，正常情况不会这么干，这里只是用来测试效果
│ └── password.txt #用来保存密码的，如果记得住密码，请忽略
├── docker-compose.yml
├── elasticsearch.yml
├── filebeat.yml
├── kibana.yml
├── logstash.conf
└── logstash.yml</code></pre>
<h2>更改目录权限</h2>
<pre><code class="language-bash">chmod -R 777 elk/data</code></pre>
<h2>vim /etc/hosts</h2>
<pre><code class="language-bash">39.108.214.196 docker-es docker-kibana docker-logstash docker-filebeat</code></pre>
<h2>1.docker-compose.yml</h2>
<pre><code class="language-bash">version: "3"
services:
es:
image: elasticsearch:8.1.1
labels:
co.elastic.logs/enabled: "false"
hostname: docker-es
ports:
- "9200:9200"
- "9300:9300"
environment:
- discovery.type=single-node
- "ES_JAVA_OPTS=-Xms256m -Xmx256m" #根据主机内存大小而定
volumes:
- ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./data/elasticsearch:/usr/share/elasticsearch/data
kibana:
image: kibana:8.1.1
labels:
co.elastic.logs/enabled: "false"
hostname: docker-kibana
ports:
- "5601:5601"
volumes:
- ./kibana.yml:/usr/share/kibana/config/kibana.yml
depends_on:
- es
logstash:
image: logstash:8.1.1
hostname: docker-logstash
ports:
- "5044:5044"
- "9600:9600"
volumes:
- ./logstash.yml:/usr/share/logstash/config/logstash.yml
- ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
- ./data/logs:/logs
environment:
- "ES_JAVA_OPTS=-Xms128m -Xmx128m" #是es的一半
- "LS_OPTS=--config.reload.automatic"
depends_on:
- es
filebeat:
image: docker.elastic.co/beats/filebeat:8.1.2
labels:
co.elastic.logs/enabled: "false"
user: root
hostname: docker-filebeat
volumes:
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml
- "/var/lib/docker/containers:/var/lib/docker/containers:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
depends_on:
- es
</code></pre>
<h2>2.elasticsearch.yml</h2>
<pre><code class="language-bash">network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false
</code></pre>
<h2>3.kibana.yml</h2>
<pre><code class="language-bash">server:
host: "0.0.0.0"
port: 5601
#ES
elasticsearch:
hosts: ["http://docker-es:9200"]
username: "kibana_system"
password: "*******" #密码需要稍后再填
#Chinese
i18n.locale: "zh-CN"</code></pre>
<h2>4.logstash.yml</h2>
<pre><code class="language-bash">http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline/*.conf
xpack.monitoring.enabled: false</code></pre>
<h2>5.logstash.conf</h2>
<pre><code class="language-bash">input {
file {
path => "/logs/*/*.log"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => [ "http://docker-es:9200" ]
user => "elastic"
password => "*******"
index => "logstash-%{+YYYY-MM-dd}"
}
}</code></pre>
<h2>6.filebeat.yml</h2>
<pre><code class="language-bash">filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
processors:
- add_cloud_metadata: ~
output.elasticsearch:
hosts: 'docker-es:9200'
username: 'elastic'
password: '*******'</code></pre>
<h2>7.初始化密码</h2>
<pre><code class="language-bash">docker-compose exec -T es elasticsearch-setup-passwords auto --batch
Tips:
然后把对应的密码填入到kibana.yml、logstash.conf、filebeat.yml中去</code></pre>
<h2>8.运行docker-compose</h2>
<pre><code class="language-bash">docker-compose up -d</code></pre>
<h2>9.测试登录</h2>
<p><a href="http://39.108.214.196:5601/">http://39.108.214.196:5601/</a> 输入用户名密码
<img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=44d129d5abf0ab44755dfd4bf303008b&file=file.png" alt="" />
39.108.214.196:9200 输入用户名和密码之后显示
<img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=fec73fe1c9ad8c5c6ffcf9d71737cd07&file=file.png" alt="" /></p>
<h1>elastic操作步骤</h1>
<h2>1.选择“自己浏览”</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=a99d59752f8db7a5de56e1e9cedfd4a3&file=file.png" alt="" /></p>
<h2>2. 点击Discover，这时没有任何数据，往./data/logs/目录中添加一个XXX.log文件，内容自定义</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=d1ca257353db7aba019459b005a94873&file=file.png" alt="" /></p>
<pre><code class="language-bash">[root@test elk]# cat data/logs/test.log
hello world
python
java
c
golang
mysql
redis
and so on</code></pre>
<h2>3.进入management，点击 “数据试图”，创建数据试图</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=6e66b0a1a90b93893b4e3a5689ea6b6b&file=file.png" alt="" />
<img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=405e3f47a7719d3f55b2ec2d942ef0de&file=file.png" alt="" /></p>
<h2>4.查询输入 logstash-</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=88c79c50d7f50f5802cc8b2930663ecc&file=file.png" alt="" /></p>
<h2>5.查看数据日志</h2>
<p><img src="https://www.showdoc.com.cn/server/api/attachment/visitFile?sign=9b57bdc2eeda9a666eb7dec77c2c59db&file=file.png" alt="" /></p>