OpenEBS Local Volume

Local PV hostpath

对比Kubernetes Hostpath卷相比，OpenEBS本地PV Hostpath卷具有以下优势:

OpenEBS本地PV Hostpath允许您的应用程序通过StorageClass、PVC和PV访问Hostpath。这为您提供了更改PV提供者的灵活性，而无需重新设计应用程序YAML
使用Velero备份和恢复进行数据保护
通过对应用程序YAML和pod完全屏蔽主机路径来防范主机路径安全漏洞

挂载分区

创建 1T 的 LVM 挂载到 /var/openebs/local 目录（local pv hostpath 默认使用目录，也可以创建新的 StorageClass 指定新的目录）

1
2


lvcreate -n lv_openebs -L 1T datavg ;mkfs.xfs /dev/datavg/lv_openebs ;mkdir -p /var/openebs/local
mount /dev/datavg/lv_openebs /var/openebs/local

安装

1

kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

默认创建两个 storageclass ，我们主要使用 openebs-hostpath

1
2
3
4


# kubectl get sc
NAME               PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-device     openebs.io/local   Delete          WaitForFirstConsumer   false                  5m43s
openebs-hostpath   openebs.io/local   Delete          WaitForFirstConsumer   false                  5m43s 

验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# vim local-hostpath-sample.yaml
---
#创建 pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-hostpath-pvc
spec:
  storageClassName: openebs-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
---
# 创建 pod
apiVersion: v1
kind: Pod
metadata:
  name: hello-local-hostpath-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-hostpath-pvc
  containers:
  - name: hello-container
    image: dockerhub.kubekey.local/library/busybox:latest
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage

kubectl apply -f local-hostpath-sample.yaml

查看 pv 和 pvc

1
2
3
4
5
6
7


# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                               STORAGECLASS       REASON   AGE
pvc-a500bbdc-65d8-4b38-a342-c15828e4a358   5G         RWO            Delete           Bound    default/local-hostpath-pvc                                                                                          openebs-hostpath            6m50s

# kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
local-hostpath-pvc   Bound    pvc-a500bbdc-65d8-4b38-a342-c15828e4a358   5G         RWO            openebs-hostpath   8m20s

对应节点的 /var/openebs/local 目录

1
2
3
4
5
6
7
8
9


# pod 路径
# kubectl exec -ti hello-local-hostpath-pod -- cat  /mnt/store/greet.txt
Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.
Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.

# 主机路径
# cat /var/openebs/local/pvc-a500bbdc-65d8-4b38-a342-c15828e4a358/greet.txt
Fri May 27 07:51:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.
Fri May 27 07:56:13 UTC 2022 [hello-local-hostpath-pod] Hello from OpenEBS Local PV.

清理 pod

1

kubectl delete pod hello-local-hostpath-pod

性能测试

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1
2
3
4


volumes:
      - name: dbench-pv
        persistentVolumeClaim:
          claimName: local-hostpath-pvc    # 修改这里

创建应用

1

kubectl create -f fio-deploy.yaml

查看状态

1
2
3


# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
dbench-zp7th--1-xjj8t   1/1     Running   0          5s

查看性能测试结果

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl logs dbench-zp7th--1-xjj8t
……
All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 130k/25.3k. BW: 2838MiB/s / 978MiB/s
Average Latency (usec) Read/Write: 93.60/384.73
Sequential Read/Write: 4128MiB/s / 1673MiB/s
Mixed Random Read/Write IOPS: 25.1k/8413

清理

1
2


kubectl delete pod dbench-zp7th--1-xjj8t
kubectl delete pvc local-hostpath-pvc

Local PV device

对比Kubernetes本地持久卷，OpenEBS本地PV设备卷有以下优点:

OpenEBS本地PV设备卷provider是动态的，Kubernetes设备卷provider是静态的
OpenEBS NDM更好地管理用于创建本地pv的块设备。 NDM提供了发现块设备属性、设置设备筛选器、度量集合以及检测块设备是否已经跨节点移动等功能

创建 SC

默认安装已经自带了 local PV 这里可以跳过

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cat > local-device-sc.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-device
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: device
      - name: FSType
        value: xfs      
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF

kubectl apply -f local-device-sc.yaml

验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# vim local-pv-sample.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-device-pvc
spec:
  storageClassName: local-device
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
---
apiVersion: v1
kind: Pod
metadata:
  name: hello-local-device-pod
spec:
  volumes:
  - name: local-storage
    persistentVolumeClaim:
      claimName: local-device-pvc
  containers:
  - name: hello-container
    image: dockerhub.kubekey.local/library/busybox:latest
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: local-storage

kubectl apply -f local-device-pod.yaml

查看 pv 和 pvc

1
2
3
4
5
6
7


# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
hello-local-device-pod   1/1     Running   0          10m

# kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
local-device-pvc   Bound    pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce   5G         RWO            local-device   10m

查看磁盘（ndm会自动选择合适的数据盘）

如下所示，sdb 被使用了，自动选择了 sdc 创建分区挂载到 pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


NAME                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                   8:0    0 446.1G  0 disk
├─sda1                8:1    0     1G  0 part /boot/efi
├─sda2                8:2    0     1G  0 part /boot
└─sda3                8:3    0 444.1G  0 part
  ├─rootvg-lv_root  252:0    0    20G  0 lvm  /
  ├─rootvg-lv_swap  252:1    0    16G  0 lvm
  ├─rootvg-lv_var   252:2    0    60G  0 lvm  /var
  ├─rootvg-lv_home  252:3    0    20G  0 lvm  /home
  └─rootvg-lv_etcd  252:6    0   100G  0 lvm  /var/lib/etcd
sdb                   8:16   0   3.5T  0 disk
├─datavg-lv_docker  252:4    0   500G  0 lvm  /var/lib/docker
├─datavg-lv_kubelet 252:5    0   100G  0 lvm  /var/lib/kubelet
└─datavg-lv_openebs 252:7    0     1T  0 lvm  /var/openebs/local
sdc                   8:32   0   3.5T  0 disk /var/lib/kubelet/pods/bc10974f-1a8a-433f-84b9-b20d8dc908c4/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce

性能测试

创建 pvc 用于性能测试

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


cat > dbench-pvc.yaml <<EOF
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dbench
spec:
  storageClassName: local-device
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
EOF

kubectl apply -f dbench-pvc.yaml

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1
2
3
4


volumes:
      - name: dbench-pv
        persistentVolumeClaim:
          claimName: dbench    # 修改这里

创建应用

1

kubectl create -f fio-deploy.yaml

查看状态

1
2
3


# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
dbench-qzczk--1-kz6wf    1/1     Running   0          10s

查看磁盘挂载信息

自动选择了另外一台机器的 sdc 创建分区挂载到 pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


10.188.100.99 | CHANGED | rc=0 >>
NAME                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                   8:0    0 446.1G  0 disk
├─sda1                8:1    0     1G  0 part /boot/efi
├─sda2                8:2    0     1G  0 part /boot
└─sda3                8:3    0 444.1G  0 part
  ├─rootvg-lv_root  252:0    0    20G  0 lvm  /
  ├─rootvg-lv_swap  252:1    0    16G  0 lvm
  ├─rootvg-lv_var   252:2    0    60G  0 lvm  /var
  ├─rootvg-lv_home  252:3    0    20G  0 lvm  /home
  └─rootvg-lv_etcd  252:6    0   100G  0 lvm  /var/lib/etcd
sdb                   8:16   0   3.5T  0 disk
├─datavg-lv_docker  252:4    0   500G  0 lvm  /var/lib/docker
├─datavg-lv_kubelet 252:5    0   100G  0 lvm  /var/lib/kubelet
└─datavg-lv_openebs 252:7    0     1T  0 lvm  /var/openebs/local
sdc                   8:32   0   3.5T  0 disk /var/lib/kubelet/pods/0ebab420-cdfa-4421-8a50-3fcdbe59ac94/volumes/kubernetes.io~local-volume/pvc-c2206bd7-dbd9-45d2-ba4b-36224edbd0e4

……

10.188.100.101 | CHANGED | rc=0 >>
NAME                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                   8:0    0 446.1G  0 disk
├─sda1                8:1    0     1G  0 part /boot/efi
├─sda2                8:2    0     1G  0 part /boot
└─sda3                8:3    0 444.1G  0 part
  ├─rootvg-lv_root  252:0    0    20G  0 lvm  /
  ├─rootvg-lv_swap  252:1    0    16G  0 lvm
  ├─rootvg-lv_var   252:2    0    60G  0 lvm  /var
  ├─rootvg-lv_home  252:3    0    20G  0 lvm  /home
  └─rootvg-lv_etcd  252:6    0   100G  0 lvm  /var/lib/etcd
sdb                   8:16   0   3.5T  0 disk
├─datavg-lv_docker  252:4    0   500G  0 lvm  /var/lib/docker
├─datavg-lv_kubelet 252:5    0   100G  0 lvm  /var/lib/kubelet
└─datavg-lv_openebs 252:7    0     1T  0 lvm  /var/openebs/local
sdc                   8:32   0   3.5T  0 disk /var/lib/kubelet/pods/e8ba48b4-957d-4d6f-bc5f-d850f058d603/volumes/kubernetes.io~local-volume/pvc-f8dcfaa4-2d82-46c8-850d-8d92403aa7ce

查看性能测试结果

性能与 hostpath 模式相差不大

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl logs dbench-qzczk--1-kz6wf
……
All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 149k/25.7k. BW: 2474MiB/s / 997MiB/s
Average Latency (usec) Read/Write: 94.81/384.69
Sequential Read/Write: 4428MiB/s / 1821MiB/s
Mixed Random Read/Write IOPS: 22.4k/7434

清理

1
2


kubectl delete pod dbench-qzczk--1-kz6wf hello-local-device-pod
kubectl delete pvc dbench local-device-pvc

LVM Local PV

安装

1

kubectl apply -f https://openebs.github.io/charts/lvm-operator.yaml

查看 pod 状态

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# kubectl get pod -A | grep openebs
kube-system            openebs-lvm-controller-0                      5/5     Running   0          74s
kube-system            openebs-lvm-node-47nz2                        2/2     Running   0          74s
kube-system            openebs-lvm-node-7nl47                        2/2     Running   0          74s
kube-system            openebs-lvm-node-dwrkc                        2/2     Running   0          74s
kube-system            openebs-lvm-node-g55vx                        2/2     Running   0          74s
kube-system            openebs-lvm-node-gxwgr                        2/2     Running   0          74s
kube-system            openebs-lvm-node-hkz69                        2/2     Running   0          74s
kube-system            openebs-lvm-node-kv48x                        2/2     Running   0          74s
kube-system            openebs-lvm-node-mqwm8                        2/2     Running   0          74s
kube-system            openebs-lvm-node-tsxsl                        2/2     Running   0          74s
kube-system            openebs-lvm-node-z2z6g                        2/2     Running   0          74s

创建 vg

集群中部分机器有 sdc ，手动在 sdc 上面创建 lvmvg 用于 openebs lvm local pv

1
2


pvcreate /dev/sdc
vgcreate lvmvg /dev/sdc

创建 SC

因为 vg 只在部分机器上面有，所以需要使用 allowedTopologies 选项，指定可以使用的节点

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


cat > lvm-local-pv-sc.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvmpv
allowVolumeExpansion: true
parameters:
  storage: "lvm"
  volgroup: "lvmvg"  #上面创建的 vg 名
provisioner: local.csi.openebs.io
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:   #有 lvmvg 的节点
      - wx-vpck8s-worker098
      - wx-vpck8s-worker099
      - wx-vpck8s-worker100
      - wx-vpck8s-worker101
      - wx-vpck8s-worker102
EOF

kubectl apply -f lvm-local-pv-sc.yaml

部署应用

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


cat > lvm-local-pv-example.yaml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: csi-lvmpv
spec:
  storageClassName: openebs-lvmpv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: lvm-pod
spec:
  restartPolicy: Never
  containers:
  - name: perfrunner
    image: dockerhub.kubekey.local/library/busybox:latest
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
       - mountPath: /datadir
         name: lvm-vol
    tty: true
  volumes:
  - name: lvm-vol
    persistentVolumeClaim:
      claimName: csi-lvmpv
EOF

kubectl apply -f lvm-local-pv-example.yaml

查看应用和lvm信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# kubectl get pod,pv,pvc
NAME          READY   STATUS    RESTARTS   AGE
pod/lvm-pod   1/1     Running   0          27s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS    REASON   AGE
persistentvolume/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e   10Gi       RWO            Delete           Bound    default/csi-lvmpv   openebs-lvmpv            26s

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/csi-lvmpv   Bound    pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e   10Gi       RWO            openebs-lvmpv   27s

# lvm 信息
NAME                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdc                                                     8:32   0   3.5T  0 disk
└─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8    0    10G  0 lvm  /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount

性能测试

创建新的 pvc 用于性能测试

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


cat > dbench-pvc.yaml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dbench-lvm
spec:
  storageClassName: openebs-lvmpv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
EOF

kubectl apply -f dbench-pvc.yaml

下载性能测试文件：

performance-benchmark/fio-deploy.yaml at master · openebs/performance-benchmark

调整以下内容：

1
2
3
4


volumes:
      - name: dbench-lvm
        persistentVolumeClaim:
          claimName: dbench    # 修改这里

创建应用

1

kubectl create -f fio-deploy.yaml

查看状态

1
2
3
4


# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
dbench-729c9--1-4clvm   1/1     Running   0          5s
lvm-pod                 1/1     Running   0          6m2s

查看磁盘挂载信息

自动选择了另外一台机器的 sdc 创建分区挂载到 pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


 10.188.100.99 | CHANGED | rc=0 >>
NAME                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                                     8:0    0 446.1G  0 disk
├─sda1                                                  8:1    0     1G  0 part /boot/efi
├─sda2                                                  8:2    0     1G  0 part /boot
└─sda3                                                  8:3    0 444.1G  0 part
  ├─rootvg-lv_root                                    252:0    0    20G  0 lvm  /
  ├─rootvg-lv_swap                                    252:1    0    16G  0 lvm
  ├─rootvg-lv_var                                     252:2    0    60G  0 lvm  /var
  ├─rootvg-lv_home                                    252:3    0    20G  0 lvm  /home
  └─rootvg-lv_etcd                                    252:6    0   100G  0 lvm  /var/lib/etcd
sdb                                                     8:16   0   3.5T  0 disk
├─datavg-lv_docker                                    252:4    0   500G  0 lvm  /var/lib/docker
├─datavg-lv_kubelet                                   252:5    0   100G  0 lvm  /var/lib/kubelet
└─datavg-lv_openebs                                   252:7    0     1T  0 lvm  /var/openebs/local
sdc                                                     8:32   0   3.5T  0 disk
└─lvmvg-pvc--763113e0--c96f--4487--b3ae--9a306c159231 252:8    0    50G  0 lvm  /var/lib/kubelet/pods/61686ab9-0538-4a89-a8e7-84ce9af351b1/volumes/kubernetes.io~csi/pvc-763113e0-c96f-4487-b3ae-9a306c159231/mount

10.188.100.98 | CHANGED | rc=0 >>
NAME                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                                     8:0    0 446.1G  0 disk
├─sda1                                                  8:1    0     1G  0 part /boot/efi
├─sda2                                                  8:2    0     1G  0 part /boot
└─sda3                                                  8:3    0 444.1G  0 part
  ├─rootvg-lv_root                                    252:0    0    20G  0 lvm  /
  ├─rootvg-lv_swap                                    252:1    0    16G  0 lvm
  ├─rootvg-lv_var                                     252:2    0    60G  0 lvm  /var
  ├─rootvg-lv_home                                    252:3    0    20G  0 lvm  /home
  └─rootvg-lv_etcd                                    252:6    0   100G  0 lvm  /var/lib/etcd
sdb                                                     8:16   0   3.5T  0 disk
├─datavg-lv_docker                                    252:4    0   500G  0 lvm  /var/lib/docker
├─datavg-lv_kubelet                                   252:5    0   100G  0 lvm  /var/lib/kubelet
└─datavg-lv_openebs                                   252:7    0     1T  0 lvm  /var/openebs/local
sdc                                                     8:32   0   3.5T  0 disk
└─lvmvg-pvc--9abc3a59--cefa--4c78--8189--ddc2ee09e58e 252:8    0    10G  0 lvm  /var/lib/kubelet/pods/32dd581a-e163-4fe1-a8a5-a1eafb396d71/volumes/kubernetes.io~csi/pvc-9abc3a59-cefa-4c78-8189-ddc2ee09e58e/mount

查看性能测试结果

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


kubectl logs dbench-729c9--1-4clvm
……
All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 125k/23.9k. BW: 2623MiB/s / 962MiB/s
Average Latency (usec) Read/Write: 95.08/385.63
Sequential Read/Write: 5042MiB/s / 1819MiB/s
Mixed Random Read/Write IOPS: 23.7k/7938

清理

1
2
3


kubectl delete -f fio-deploy.yaml
kubectl delete -f dbench-pvc.yaml
kubectl delete -f lvm-local-pv-example.yaml

总结

三种模式pod磁盘性能差不多，lvm 模式下顺序读性能稍好。

Local PV hostpath 需要手动挂载磁盘到对应目录
Local PV device pod独占磁盘
LVM Local PV 需要手动创建vg，pod 使用 lvm 较为灵活，目前推荐使用

参考文章：

https://weiliang-ms.github.io/wl-awesome/2.%E5%AE%B9%E5%99%A8/k8s/storage/OpenEBS.html

https://github.com/openebs/lvm-localpv

Nickname

Website

0/500

OωO
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
（╯‵□′）╯︵┴─┴
￣﹃￣
(/ω＼)
∠( ᐛ 」∠)＿
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ｀)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ(￣∇￣o)
ヾ(´･･｀｡)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò｡)
Σ(っ °Д °;)っ
( ,,´･ω･)ﾉ"(´っω･｀｡)
╮(╯▽╰)╭
o(*////▽////*)q
＞﹏＜
( ๑´•ω•) "(ㆆᴗㆆ)

😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣

颜文字
Emoji
Bilibili

0 comments

No comment