MinIO分布式存储方案预研

原创阿虎
发布于:2022-04-20 
标签: 

MinIO简介

MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口,非常适合于存储大容量非结构化的数据,例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等,而一个对象文件可以是任意大小,从几kb到最大5T不等。

MinIO是一个非常轻量的服务,可以很简单的和其他应用的结合,类似 NodeJS, Redis 或者 MySQL

http://docs.minio.org.cn/docs/
https://docs.min.io/docs/minio-quickstart-guide.html

优点

缺点

社区版本免费

file

安装

docker run -d -p 9000:9000 --name minio \
  -v /media/hushow/doc/minio-data/data:/data \
  -v /media/hushow/doc/minio-data/config:/root/.minio \
  minio/minio server /data --console-address ":9001"

启动成功
file

控制台

基于临时凭证流程

file

控制台申请永久凭证(配好权限)

file

后端JAVA-SDK上传

官方示例 https://github.com/minio/minio-java/tree/release/examples

前端JS-SDK上传

官方示例 https://github.com/minio/minio-js/tree/master/examples

  • 从后台获取临时凭证
    @Test
    public void getTempCredentialsTest() throws Exception {
        String bucketName = "demo";
        String ak = "AJEDFDFDFDFDFDF";
        String sk="SJEDFDFDFDFDKKK";
        String endpoint = "http://172.17.0.2:9000";

        Provider provider =
                new AssumeRoleProvider(
                        endpoint, // STS endpoint usually point to MinIO server.
                        ak, // Access key.
                        sk, // Secret key.
                        2000, // Duration seconds if available.
                        null, // Policy if available.
                        null, // Region if available.
                        null, // Role ARN if available.
                        null, // Role session name if available.
                        null, // External ID if available.
                        null);

        Credentials credential = provider.fetch();
        log.info("ak:{} sk:{}, st:{}", credential.accessKey(), credential.secretKey(), credential.sessionToken());
    }

js-sdk浏览器上传问题:

[Vue warn]: Error in v-on handler: "TypeError: _fs.default.stat is not a function"

found in

---> <Upload> at packages/upload/src/upload.vue
       <ElUpload> at packages/upload/src/index.vue
         <MinioUpload> at src/views/minio-upload.vue
           <App> at src/App.vue
             <Root>
warn @ vue.runtime.esm.js:619
logError @ vue.runtime.esm.js:1884
globalHandleError @ vue.runtime.esm.js:1879
handleError @ vue.runtime.esm.js:1839
invokeWithErrorHandling @ vue.runtime.esm.js:1862
invoker @ vue.runtime.esm.js:2179
original._wrapper @ vue.runtime.esm.js:6917
vue.runtime.esm.js:1888 TypeError: _fs.default.stat is not a function
    at _async.default.waterfall.size (minio.js:1148)
    at nextTask (async.mjs:5781)
    at Object.waterfall (async.mjs:5792)
    at Object.awaitable [as waterfall] (async.mjs:205)
    at Client.fPutObject (minio.js:1148)
    at Client.<anonymous> (helpers.js:97)
    at VueComponent.handleUpload (minio-upload.vue:106)
    at VueComponent.post (element-ui.common.js:29381)
    at VueComponent.upload (element-ui.common.js:29336)
    at element-ui.common.js:29300

file

排查发现,mini js-sdk只能在后端跑:

https://github.com/minio/minio-js/issues/858

mini在浏览器使用方案minio-js-browser:

https://github.com/minio/minio-js/issues/855
https://codesandbox.io/s/react-minio-js-example-9xtgo
https://github.com/harshavardhana/minio-js-browser-upload
https://github.com/prakashsvmx/minio-js-web-browser-example

运行minio-js-web-browser-example代码,大文件分片上传效果如下:

file

s3 js-sdk替代方案

https://github.com/davideliason/s3upload
https://github.com/davideliason/s3upload/blob/master/index.html
http://docs.minio.org.cn/docs/master/how-to-use-aws-sdk-for-javascript-with-minio-server


<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
<meta name="viewport" content="width=device-width,initial-scale=1.0,minimum-scale=1.0" />
<title>s3 Upload</title>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.19.0.min.js"></script>
</head>
<body>
<input type="file" id="file-chooser" />
<button id="upload-button">Upload to S3</button>
<div id="results"></div>
<script type="text/javascript">    
    // set variables for click handlers
    var fileChooser = document.getElementById('file-chooser');
    var button = document.getElementById('upload-button');
    var results = document.getElementById('results');

    button.addEventListener('click', function () {
        var file = fileChooser.files[0];
        debugger;
        if (file) {            
            AWS.config.update({
                "accessKeyId": "AJEDFDFDFDFDFDF",
                "secretAccessKey": "SJEDFDFDFDFDKKK",
                "s3ForcePathStyle": true,
                "region": "us-west-2",
                "endpoint":"http://172.17.0.2:9000",
                "signatureVersion":"v4"
            });
            var s3 = new AWS.S3();
            var params = {
                Bucket: 'demo',
                Key: file.name,
                ContentType: file.type,
                Body: file,
                ACL: 'public-read'
            };        
            s3.putObject(params, function (err, res) {
                if (err) {
                    results.innerHTML = ("Error uploading data: ", err);
                } else {
                    results.innerHTML = ("Successfully uploaded data");
                }
            });
        } else {
            results.innerHTML = 'Nothing to upload.';
        }
    }, false);
</script>

</body>
</html>