1、Pmem介绍

PMEM是硬件产品,Intel Optane DC持久存储模块,是一种具有大容量和数据持久性的创新存储技术。有2种运行模式。两级内存模式无需软件更改,DCPMM被视为更大的内存,并使用DRAM作为其缓存层。AppDirect模式将设备暴露为持久内存,支持软件栈,可用于加速不同的应用程序。在本文中,我们将使用AppDirect模式。

2、环境信息

  • 一台master和一台node构成的K8s1.18.6集群
  • redis镜像为根据源码编译为pmem-redis:4.0.0
  • ceph/neonsan块存储
  • centos7.7/内核5.8.7

3、测试两种模式

  • 测试目的是体验PMEM对REDIS的加速效果。先在没有PMEM加速AOF模式,再有PMEM加速PBA模式。
  • AOF模式是append only file的意思,通常REDIS是一种内存数据库,数据掉电就丢失了。AOF模式可以把数据库记录随时备份到分布式存储里,这样可以使得REDIS具有掉电恢复的功能。
  • PBA模式是pointer based AOF模式,它是使用PMEM对AOF做了加速,原理是备份写盘时只把指针写到磁盘里,数据还在内存或PMEM里,使用PMEM作为缓存。这样既可以掉电恢复,又提升了性能。充分发挥了PMEM AD模式的优势。

4、pmem-redis镜像构建

5、PBA模式的设置

  • ipmctl安装
cd /etc/yum.repos.d/
wget https://copr.fedorainfracloud.org/coprs/jhli/ipmctl/repo/epel-7/jhli-ipmctl-epel-7.repo
wget https://copr.fedorainfracloud.org/coprs/jhli/safeclib/repo/epel-7/jhli-safeclib-epel-7.repo
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
el-ha-for-rhel-*-server-rpms"
yum install ndctl ndctl-libs ndctl-devel libsafec rubygem-asciidoctor
yum install ipmctl
  • app direct模式
sudo ipmctl delete -goal
sudo ipmctl create -goal PersistentMemoryType=AppDirect

A reboot is required to process new memory allocation goals:
sudo reboot
  • 检测Pmem能正常工作且为ad模式,ad是否有值
ipmctl show -memoryresources
 MemoryType   | DDR         | PMemModule  | Total
========================================================
 Volatile     | 191.000 GiB | 0.000 GiB   | 191.000 GiB
 AppDirect    | -           | 504.000 GiB | 504.000 GiB
 Cache        | 0.000 GiB   | -           | 0.000 GiB
 Inaccessible | 1.000 GiB   | 1.689 GiB   | 2.689 GiB
 Physical     | 192.000 GiB | 505.689 GiB | 697.689 GiB

 当ad没有值时,ipmctl start -diagnostic诊断是否有错误消息
 刚开始遇到这样的一个问题: 
 The platform configuration check detected that PMem module 0x0001 is not configured.
 分析为:新版Ipmctl有问题,用1.x的版本把pcd delete以后重新provision就可以了

6、Pmem-csi安装

下载, https://github.com/intel/pmem-csi/blob/devel/docs/install.md#install-pmem-csi-driver
cd pmem-CSI

Setting up certificates for securities
# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o _work/bin/cfssl --create-dirs
# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o _work/bin/cfssljson --create-dirs
# chmod a+x _work/bin/cfssl _work/bin/cfssljson
# export PATH=$PATH:$PWD/_work/bin
# ./test/setup-ca-kubernetes.sh

Deploying the driver to K8s using LVM mode, please choose yaml files corresponding to your kubernetes version
# kubectl create -f deploy/kubernetes-1.18/pmem-csi-lvm.yaml
Applying a storage class
# kubectl apply -f deploy/kubernetes-1.18/pmem-storageclass-ext4.yaml
pod状态
kubectl get pod
NAME                    READY   STATUS        RESTARTS   AGE
pmem-csi-controller-0   2/2     Running       0          22s
pmem-csi-node-tw4mw     2/2     Running       2          33h
sc状态
kubectl get sc
NAME               PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
pmem-csi-sc-ext4   pmem-csi.intel.com         Delete          Immediate           false                  3d2h

7、benchmark安装

  • yum install autoconf automake make gcc-c++
  • yum install pcre-devel zlib-devel libmemcached-devel

Remove system libevent and install new version:

Build and install memtier:

8、测试对比

  • aof的yanl文件
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: csi-neonsan
---
kind: Pod
apiVersion: v1
metadata:
  name: redis-aof
  labels:
    app: redis-aof
spec:
  containers:
    - name: redis-aof
      image: pmem-redis:latest
      imagePullPolicy: IfNotPresent
      args: ["no"]
      ports:
      - containerPort: 6379
      resources:
        limits:
          cpu: "1000m"
          memory: "16Gi"
        requests:
          cpu: "1000m"
          memory: "16Gi"
      volumeMounts:
      - mountPath: "/ceph"
        name: ceph-csi-volume
  volumes:
  - name: ceph-csi-volume
    persistentVolumeClaim:
      claimName: rbd-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis-aof
spec:
  type: NodePort
  ports:
  - name: redis
    port: 6379
    nodePort: 30379
    targetPort: 6379
  selector:
    app: redis-aof
  • pba的yanl文件
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-csi-pvc-ext4
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: pmem-csi-sc-ext4 # defined in pmem-storageclass-ext4.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: csi-neonsan
---
kind: Pod
apiVersion: v1
metadata:
  name: redis-with-pba
  labels:
    app: redis-with-pba
spec:
  containers:
    - name: redis-with-pba
      image: pmem-redis:latest
      imagePullPolicy: IfNotPresent
      args: ["yes"]
      ports:
      - containerPort: 6379
      resources:
        limits:
          cpu: "1000m"
          memory: "16Gi"
        requests:
          cpu: "1000m"
          memory: "16Gi"
      volumeMounts:
      - mountPath: "/data"
        name: my-csi-volume
      - mountPath: "/ceph"
        name: ceph-csi-volume
  volumes:
  - name: ceph-csi-volume
    persistentVolumeClaim:
      claimName: rbd-pvc
  - name: my-csi-volume
    persistentVolumeClaim:
      claimName: pmem-csi-pvc-ext4
---
apiVersion: v1
kind: Service
metadata:
  name: redis-pba
  labels:
    app: redis-with-pba
spec:
  type: NodePort
  ports:
  - name: redis
    port: 6379
    nodePort: 31379
    targetPort: 6379
  selector:
    app: redis-with-pba
  • 先执行aof的yaml文件,kubectl apply -f aof.yaml,然后再执行memtier_benchmark
memtier_benchmark -s 172.30.35.9  -p 30379 -R -d 1024 --key-maximum=1000000 -n 10000  --ratio=1:9 | grep Totals>>aof_1024
Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  37 secs]  0 threads:     2000000 ops,   53289 (avg:   53544) ops/sec, 7.23MB/sec (avg: 7.29MB/sec),  3.75 (avg:  3.73) msec latency



[root@neonsan-10 scripts]# cat aof_1024
Totals      53508.29        26.75     48130.71         3.73350         3.27900         8.31900        18.43100      7456.87
  • 先执行pba的yaml文件,kubectl apply -f pba.yaml,然后再执行memtier_benchmark,注意yaml文件里面同时挂载不同存储。
memtier_benchmark -s 172.30.35.9  -p 31379 -R -d 1024 --key-maximum=1000000 -n 10000  --ratio=1:9 | grep Totals>>pba_1024
Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  24 secs]  0 threads:     2000000 ops,   81619 (avg:   81294) ops/sec, 11.07MB/sec (avg: 11.10MB/sec),  2.45 (avg:  2.46) msec latency


[root@neonsan-10 scripts]# cat pba_1024
Totals          0.00         0.00         0.00            -nan         0.00000         0.00000         0.00000         0.00
Totals      82856.35        82.86     74487.86         2.45929         2.38300         4.79900         6.30300     11588.37
  • 通过速度和延迟性比较两种存储的性能。