MinIO大型分布式对象存储集群部署

一、介绍

实验环境：

服务器	IP
MinIO集群节点1	192.168.168.101
MinIO集群节点2	192.168.168.102
MinIO集群节点3	192.168.168.103
MinIO集群节点4	192.168.168.104
Nginx	192.168.168.105
MinIO客户端	192.168.168.106

服务器规划表

二、服务端安装部署

（1）先关闭防火墙、selinx、NetworkManager
useradd -M -r minio-user
yum -y install minio
（2）添加存储设备，并格式化挂载，以便作为minio的共享设备使用
mkfs.xfs /dev/sdb
mkdir /mnt/disk
mount /dev/sdb /mnt/disk
chown minio-user.minio-user /mnt/disk
（3）配置文件
vi /etc/default/minio
MINIO_VOLUMES="http://192.168.168.{101...104}:9000/mnt/disk{1...2}/minio"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
~
-------------------------end-------------------------------------
vi /usr/lib/systemd/system/minio.service
#ProtectProc=invisible
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
-------------------------end-------------------------------------
(4)启动minio服务
systemctl daemon-reload
systemctl start minio

注意：如果minio集群节点的IP地址不连续则MINIO_VOLUMES属性的设置可以使用连续主机名的方式表示。如：http://minio{1...4}.com:9000/mnt/disk{1...4}/minio"

打开web管理页

http://192.168.168.101:9001/ 用户密码在/etc/default/minio文件中有设置。

创建桶：

创建秘钥：

三、客户端安装部署

yum install perl-core zlib-devel
tar xzvf openssl-3.3.0.tar.gz
cd openssl-3.3.0
./config --prefix=/usr/local/ssl3.3 --openssldir=/usr/local/ssl3.3 shared zlib
make && make install
mkdir /usr/local/ssl3.3/lib
cp ./*.{so,so.3*,a,pc} /usr/local/ssl3.3/lib
vi /etc/ld.so.conf
……
/usr/local/ssl3.3/lib64
—————————————————end——————————————————-
ldconfig
（2）编译安装python3.12.3
tar zxvf Python-3.12.3.tgz
cd Python-3.12.3
./configure --with-openssl=/usr/local/ssl3.3 --prefix=/usr/local/python3.12
make && make install
ln -s /usr/local/python3.12/bin/python3.12 /usr/bin/python3
ln -s /usr/local/python3.12/bin/pip3.12 /usr/bin/pip3
（3）安装MinIO客户端
wget https://dl.minio.org.cn/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/local/bin/mc

客户端测试

#通过脚步上传单个文件

pip3 install minio
(如因网络问题无法下载可以把下载好的主机中的/usr/local/python3.12/lib/python3.12/site-packages文件夹拷贝到无法下载的主机既可运行)
#cd /software
echo 'dsjkldsjdjsf' > sss.txt
vi single-put.py
#!/usr/bin/python3
from minio import Minio
import os
# 创建 MinIO 客户端
client = Minio('192.168.168.101:9000',
 access_key='zBgOhND6C75474KMIKXP',
 secret_key='9frtFi7VwuaWW2VAqBUMYN9cocbxjkgxsxQVIrcb',
 secure=False)
# 列出存储桶
buckets = client.list_buckets()
for bucket in buckets:
 print(bucket.name)
# 上传对象到 MinIO
bucket_name = "bucket.qingmei01"
object_name = "sss.txt"
local_file_path = "/software/sss.txt"
# 替换为您希望上传的文件路径
# 确保本地文件存在
if not os.path.isfile(local_file_path):
 raise FileNotFoundError(f"The file {local_file_path} does not exist")
# 获取文件大小
file_size = os.path.getsize(local_file_path)
# 上传文件
with open(local_file_path, "rb") as f:
 client.put_object(bucket_name, object_name, f, file_size,
content_type="application/octet-stream")
print(f"{object_name} has been uploaded to {bucket_name}")

通过客户端操作文件：

创建别名：
[root@localhost ~]# mc alias set minio_data_A http://192.168.168.101:9000 minioadmin minioadmin
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
Added `minio_data_A` successfully.

查看系统内已有的mc别名：
mc alias list

下载单个文件：
[root@localhost ~]# mc get minio_data_A/class1/sss.txt .
...68.168.101:9000/class1/sss.txt: 13 B / 13 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 1.42 KiB/s 0s
下载文件夹：
[root@localhost ~]# mc cp --recursive  minio_data_A .
...68.168.101:9000/class1/sss.txt: 26.70 MiB / 26.70 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 116.51 MiB/s 0s
上传单个文件：
[root@localhost tmp]# mc put sss.txt minio_data_A/class2
/tmp/sss.txt:                      13 B / 13 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 1.51 KiB/s 0s
上传多个文件：
[root@localhost tmp]# mc cp --recursive /etc/default/ minio_data_A/class2
/etc/default/useradd:              2.10 KiB / 2.10 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 54.52 KiB/s 0s

四、Nginx服务器反向代理MinIO配置

为MinIO服务器S3 API和MinIO控制台Web GUI创建或配置单独的、唯一的子域名。

例如，给定根域 example.net :

将代理请求转发到子域 minio.example.net ，该子域指向在 https://minio.local:9000 上监听的MinIO服务器。
将请求代理到子域 console.example.net ，该子域指向在 https://minio.local:9001 上监听的MinIO控制台。

以下位置块为您提供了一个模板，以便在您的独特环境中进行进一步的自定义：

upstream minio_s3 {
   least_conn;
   server 192.168.168.101:9000;
   server 192.168.168.102:9000;
   server 192.168.168.103:9000;
   server 192.168.168.104:9000;
}

upstream minio_console {
   least_conn;
   server 192.168.168.101:9001;
   server 192.168.168.102:9001;
   server 192.168.168.103:9001;
   server 192.168.168.104:9001;
}

server {
   listen       80;
   listen  [::]:80;
   server_name  minio.example.net;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance
   }
}

server {

   listen       80;
   listen  [::]:80;
   server_name  console.example.net;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websocket
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      chunked_transfer_encoding off;

      proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance
   }
}

MinIO大型分布式对象存储集群部署

一、介绍

二、服务端安装部署

三、客户端安装部署

四、Nginx服务器反向代理MinIO配置

honeyshan

Leave a Reply 取消回复

分类

归档