libcopp接入C++20 Coroutine和一些过渡期的设计
author: owent categories:
Article
Blablabla
date: 2020-05-22 15:36:58
draft: false
id: 2004
tags:
tags:
libcopp
coroutine
cxx
c++20
co_await
await
rust
future
promise
async
title: libcopp接入C++20 Coroutine和一些过渡期的设计
type: post
前言
C++20 协程
Rust的零开销抽象协程
大致的结构如下:
libcopp对 C++20 协程的设计抽象
为平滑迁移而设计的 future_t 、 poll_t 和 context_t
C++并没有在语言层面保证生命周期唯一性,如果为了防止误用和生命周期混乱,要么引入引用计数来保证生命周期得可靠性,要么禁止复制。我们这里为了减少不必要的开销选择了禁止复制。
提供的基本组件如下:
future_t<T>
: 用户数据层抽象,指示未来完成以后返回类型T
context_t<TPD>
: 用户调度层抽象,用于创建用户自定义类型(TPD
)的执行状态机和通过提供构造、析构等函数提供功能性接口。TPD::operator(...)
: 用户异步调用事件通知抽象,用来通知future_t<T>
的状态变化和如何变化。目前的设计中,future_t<T>::poll(context_t<TPD>)
也会自动设置context_t<TPD>
的wake行为,并且最终回调到TPD::operator(...)
看起来代码也不少,实际上就下面这个结构:
小对象优化
copp::future::result_t<成功类型,错误类型>
copp::future::result_t<成功类型,错误类型>
然后 result_t
额外提供了 result_t::make_success(...)
和 result_t::make_error(...)
来创建可以直接移动赋值给 future_t::poll_type
的简化接口。
协程任务 task_t
上面有一些数据结构是为了嵌套 co_await
而存在,自动记录了引用关系。然后在使用过程中,我们定义协程函数就很简单了:
生成器 generator_t
我们来个完整的example
下面是一个对比,假设异步调用的结果是 RpcResult
,自定义用户调度层接入类型为 RpcCall
:
对比项
异步调用系统底层创建的对象
符合 traits Future<Output=RpcResult>
的 RpcCall
future_t<RpcResult>
、context_t<RpcCall>
和用户自定数据 RpcCall
异步调用用户层接口
调用方创建 RpcCall
调用方创建 generator_t<RpcResult,RpcCall>
异步调用调度层接入
实现 RpcCall
的 poll() -> Poll<RpcResult>
接口
实现 RpcCall::operator(future, context)
Waker和Future绑定关系
由框架调度层实现
自动绑定,用户调度层可修改替换
协程组织结构
has-a 关系 ,重入有低开销,组织管理无开销
链式关系,重入几乎零开销,组织管理低开销
Context/Waker生命周期
跟随异步任务,多个异步调用可以复用同一个Waker
跟随异步调用
异步任务系统底层创建的对象
现有的实现里大部分由额外的Task对象和管理层对象
比异步调用多 task_t<RpcResult, RpcCall>
、runtime_t<RpcResult, RpcCall>
和task_t<RpcResult, RpcCall>::promise_type
异步任务用户层接口
比异步调用多一个对Future的生命周期管理(一般会包到Pin<Box<Future<Output=RpcResult>>>
里丢到堆上)
调用方函数返回 task_t<RpcResult,RpcCall>
异步任务调度层接入
非常复杂,设计多个对象的生命周期管理和调度入口管理
很简单,也可以使用 task_t<RpcResult, void>
来实现零接入
异步任务超时和错误管理
只能依赖await前嵌套timeout future实现,必须转换成相同的输出类型
允许外部总控超时和错误,对父子模块间无类型要求
多个框架融合
非常复杂,要合并多个框架的调度入口、事件管理和生命周期管理
压力测试
压力测试机器环境:
环境名称
值
系统
Linux kernel 3.10.107(Docker)
CPU
Intel(R) Xeon(R) Gold 61xx CPU @ 2.50GHz * 48
L1 Cache
64Bytes*64sets*8ways=32KB
系统负载
1.67 0.56 0.30
内存占用
23GB(used)/125GB(sum)
CMake
3.17.0
GCC版本
10.1.0
Clang版本
10.0.0
libcopp
dev分支(20200520)
librf
master分支(20200519)
组件(Avg)
协程数:1 切换开销
协程数:1000 创建开销
协程数:1000 切换开销
协程数:30000 创建开销
协程数:30000 切换开销
C++20 Coroutine - Clang
5 ns
130 ns
6 ns
136 ns
9 ns
C++20 Coroutine - GCC
7 ns
146 ns
7 ns
120 ns
9 ns
310 ns / 292 ns
252 ns / 245 ns
29 ns / 29 ns
281 ns / 229ns
33 ns / 31 ns
32 ns
96 ns
77 ns
212 ns
213 ns
49 ns
134 ns
134 ns
256 ns
371 ns
libcopp future_t(no trivial) - GCC
4 ns / 4 ns
26 ns / 24 ns
4 ns / 4 ns
26 ns / 31 ns
6 ns / 5 ns
libcopp future_t - GCC
4 ns / 4 ns
26 ns / 25 ns
4 ns / 4 ns
30 ns / 30 ns
9 ns / 5 ns
libcopp task_t(no trivial) - GCC
21 ns / 23 ns
120 ns / 118 ns
25 ns / 25 ns
122 ns / 131 ns
35 ns / 33 ns
libcopp task_t - GCC
41 ns / 32 ns
112 ns / 120 ns
41 ns / 32 ns
122 ns / 131 ns
50 ns / 38 ns
libcopp future_t(no trivial) - Clang
5 ns / 5 ns
30 ns / 30 ns
5 ns / 5 ns
30 ns / 35 ns
7 ns / 6 ns
libcopp future_t - Clang
7 ns / 7 ns
30 ns / 30 ns
7 ns / 7 ns
30 ns / 37 ns
8 ns / 8 ns
libcopp task_t(no trivial) - Clang
24 ns / 24 ns
237 ns / 142 ns
24 ns / 24 ns
357 ns / 156 ns
44 ns / 33 ns
libcopp task_t - Clang
53 ns / 44 ns
257 ns / 155 ns
53 ns / 45 ns
357 ns / 175 ns
70 ns / 61 ns
最后测出来的创建 task_t
的开销分布如下:
最后测出来的切换(创建 generator_t
)的开销分布如下:
{ "type": "bar", "data": { labels: ['协程数:1,栈大小16KB', '协程数:1000,栈大小2MB', '协程数:30000,栈大小64KB'], "datasets": [ { "label": "C++20 Coroutine - Clang 切换耗时", "borderColor": "rgba(139, 0, 0, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [5, 6, 9], "type": 'line' }, { "label": "C++20 Coroutine - MSVC 切换耗时", "borderColor": "rgba(0, 0, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [10, 14, 28], "type": 'line' }, { "label": "librf(C++20 Coroutine) - Clang 切换耗时", "borderColor": "rgba(0, 139, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [null, 29, 33], "type": 'line' }, { "label": "libcopp+动态栈池 切换耗时", "borderColor": "rgba(184, 134, 11, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [32, 77, 213], "type": 'line' }, { "label": "libcopp+libcotask+动态栈池 切换耗时", "borderColor": "rgba(189, 183, 107, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [49, 134, 371], "type": 'line' }, { "label": "libcopp future_t - GCC 切换耗时", "borderColor": "rgba(139, 0, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [4, 4, 9], "type": 'line' }, { "label": "libcopp future_t(no trivial) - GCC 切换耗时", "borderColor": "rgba(85, 107, 47, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [4, 4, 6], "type": 'line' }, { "label": "libcopp task_t - GCC 切换耗时", "borderColor": "rgba(255, 140, 0, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [23, 25, 35], "type": 'line' }, { "label": "libcopp task_t(no trivial) - GCC 切换耗时", "borderColor": "rgba(153, 50, 204, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [41, 41, 50], "type": 'line' }, { "label": "libcopp future_t - Clang 切换耗时", "borderColor": "rgba(233, 150, 122, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [5, 5, 7], "type": 'line' }, { "label": "libcopp future_t(no trivial) - Clang 切换耗时", "borderColor": "rgba(143, 188, 143, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [7, 7, 8], "type": 'line' }, { "label": "libcopp task_t - Clang 切换耗时", "borderColor": "rgba(255, 20, 147, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [24, 24, 44], "type": 'line' }, { "label": "libcopp task_t(no trivial) - Clang 切换耗时", "borderColor": "rgba(72, 61, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [53, 53, 70], "type": 'line' }, { "label": "C++20 Coroutine - GCC 创建耗时", "backgroundColor": "rgba(128, 0, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 120, 122] }, { "label": "C++20 Coroutine - Clang 创建耗时", "backgroundColor": "rgba(255, 0, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 130, 136] }, { "label": "C++20 Coroutine - MSVC 创建耗时", "backgroundColor": "rgba(0, 0, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 407, 369] }, { "label": "librf(C++20 Coroutine) - Clang 创建耗时", "backgroundColor": "rgba(0, 255, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 252, 281] }, { "label": "libcopp+动态栈池 创建耗时", "backgroundColor": "rgba(218, 165, 32, 1)", "yAxisID": 'y-axis-1', "data": [null, 96, 212] }, { "label": "libcopp+libcotask+动态栈池 创建耗时", "backgroundColor": "rgba(240, 230, 140, 1)", "yAxisID": 'y-axis-1', "data": [null, 134, 256] }, { "label": "libcopp future_t - GCC 创建耗时", "backgroundColor": "rgba(255, 0, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 26, 30] }, { "label": "libcopp future_t(no trivial) - GCC 创建耗时", "backgroundColor": "rgba(128, 128, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 26, 26] }, { "label": "libcopp task_t(no trivial) - GCC 创建耗时", "backgroundColor": "rgba(255, 165, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 120, 122] }, { "label": "libcopp task_t - GCC 创建耗时", "backgroundColor": "rgba(218, 112, 214, 1)", "yAxisID": 'y-axis-1', "data": [null, 112, 122] }, { "label": " libcopp future_t - Clang 创建耗时", "backgroundColor": "rgba(250, 128, 114, 1)", "yAxisID": 'y-axis-1', "data": [null, 30, 30] }, { "label": "libcopp future_t(no trivial) - Clang 创建耗时", "backgroundColor": "rgba(46, 139, 87, 1)", "yAxisID": 'y-axis-1', "data": [null, 30, 30] }, { "label": "libcopp task_t(no trivial) - Clang 创建耗时", "backgroundColor": "rgba(106, 90, 205, 1)", "yAxisID": 'y-axis-1', "data": [null, 237, 357] }, { "label": "libcopp task_t - Clang 创建耗时", "backgroundColor": "rgba(112, 128, 144, 1)", "yAxisID": 'y-axis-1', "data": [null, 257, 357] }] }, "options": { title: { display: true, text: '切换/创建耗时(越小越好)' }, scales: { yAxes: [{ type: 'linear', display: true, scaleLabel: { display: true, labelString: "切换耗时(单位:纳秒)" }, position: 'left', id: 'y-axis-2', gridLines: { drawOnChartArea: false }, ticks: { callback: function(value, index, values) { return value + ' ns'; } } }, { type: 'logarithmic', display: true, scaleLabel: { display: true, labelString: "创建耗时(单位:纳秒)" }, ticks: { autoSkip: true, callback: function(value, index, values) { for (var idx in values) { var tv = values[idx]; if (tv < value && Math.floor(Math.log(value)) == Math.floor(Math.log(tv))) { return null; } } return value + ' ns'; } }, position: 'right', id: 'y-axis-1', }], } } }
{ "type": "bar", "data": { labels: ['协程数:1,栈大小16KB', '协程数:1000,栈大小2MB', '协程数:30000,栈大小64KB'], "datasets": [ { "label": "C++20 Coroutine - Clang 切换耗时", "borderColor": "rgba(139, 0, 0, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [5, 6, 9], "type": 'line' }, { "label": "C++20 Coroutine - MSVC 切换耗时", "borderColor": "rgba(0, 0, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [10, 14, 28], "type": 'line' }, { "label": "librf(C++20 Coroutine) - Clang 切换耗时", "borderColor": "rgba(0, 139, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [null, 29, 31], "type": 'line' }, { "label": "libcopp+动态栈池 切换耗时", "borderColor": "rgba(184, 134, 11, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [32, 77, 213], "type": 'line' }, { "label": "libcopp+libcotask+动态栈池 切换耗时", "borderColor": "rgba(189, 183, 107, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [49, 134, 371], "type": 'line' }, { "label": "libcopp future_t - GCC 切换耗时", "borderColor": "rgba(139, 0, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [4, 4, 5], "type": 'line' }, { "label": "libcopp future_t(no trivial) - GCC 切换耗时", "borderColor": "rgba(85, 107, 47, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [4, 4, 5], "type": 'line' }, { "label": "libcopp task_t(no trivial) - GCC 切换耗时", "borderColor": "rgba(255, 140, 0, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [23, 25, 33], "type": 'line' }, { "label": "libcopp task_t - GCC 切换耗时", "borderColor": "rgba(153, 50, 204, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [32, 32, 38], "type": 'line' }, { "label": "libcopp future_t - Clang 切换耗时", "borderColor": "rgba(233, 150, 122, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [5, 5, 6], "type": 'line' }, { "label": "libcopp future_t(no trivial) - Clang 切换耗时", "borderColor": "rgba(143, 188, 143, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [7, 7, 8], "type": 'line' }, { "label": "libcopp task_t(no trivial) - Clang 切换耗时", "borderColor": "rgba(255, 20, 147, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [24, 24, 33], "type": 'line' }, { "label": "libcopp task_t - Clang 切换耗时", "borderColor": "rgba(72, 61, 139, 1)", "fill": false, "yAxisID": 'y-axis-2', "data": [44, 45, 61], "type": 'line' }, { "label": "C++20 Coroutine - GCC 创建耗时", "backgroundColor": "rgba(128, 0, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 118, 131] }, { "label": "C++20 Coroutine - Clang 创建耗时", "backgroundColor": "rgba(255, 0, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 130, 136] }, { "label": "C++20 Coroutine - MSVC 创建耗时", "backgroundColor": "rgba(0, 0, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 407, 369] }, { "label": "librf(C++20 Coroutine) - Clang 创建耗时", "backgroundColor": "rgba(0, 255, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 245, 229] }, { "label": "libcopp+动态栈池 创建耗时", "backgroundColor": "rgba(218, 165, 32, 1)", "yAxisID": 'y-axis-1', "data": [null, 96, 212] }, { "label": "libcopp+libcotask+动态栈池 创建耗时", "backgroundColor": "rgba(240, 230, 140, 1)", "yAxisID": 'y-axis-1', "data": [null, 134, 256] }, { "label": "libcopp future_t - GCC 创建耗时", "backgroundColor": "rgba(255, 0, 255, 1)", "yAxisID": 'y-axis-1', "data": [null, 25, 30] }, { "label": "libcopp future_t(no trivial) - GCC 创建耗时", "backgroundColor": "rgba(128, 128, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 24, 31] }, { "label": "libcopp task_t(no trivial) - GCC 创建耗时", "backgroundColor": "rgba(255, 165, 0, 1)", "yAxisID": 'y-axis-1', "data": [null, 118, 131] }, { "label": "libcopp task_t - GCC 创建耗时", "backgroundColor": "rgba(218, 112, 214, 1)", "yAxisID": 'y-axis-1', "data": [null, 120, 131] }, { "label": " libcopp future_t - Clang 创建耗时", "backgroundColor": "rgba(250, 128, 114, 1)", "yAxisID": 'y-axis-1', "data": [null, 30, 35] }, { "label": "libcopp future_t(no trivial) - Clang 创建耗时", "backgroundColor": "rgba(46, 139, 87, 1)", "yAxisID": 'y-axis-1', "data": [null, 30, 37] }, { "label": "libcopp task_t(no trivial) - Clang 创建耗时", "backgroundColor": "rgba(106, 90, 205, 1)", "yAxisID": 'y-axis-1', "data": [null, 142, 156] }, { "label": "libcopp task_t - Clang 创建耗时", "backgroundColor": "rgba(112, 128, 144, 1)", "yAxisID": 'y-axis-1', "data": [null, 155, 175] }] }, "options": { title: { display: true, text: '切换/创建耗时(越小越好)' }, scales: { yAxes: [{ type: 'linear', display: true, scaleLabel: { display: true, labelString: "切换耗时(单位:纳秒)" }, position: 'left', id: 'y-axis-2', gridLines: { drawOnChartArea: false }, ticks: { callback: function(value, index, values) { return value + ' ns'; } } }, { type: 'logarithmic', display: true, scaleLabel: { display: true, labelString: "创建耗时(单位:纳秒)" }, ticks: { autoSkip: true, callback: function(value, index, values) { for (var idx in values) { var tv = values[idx]; if (tv < value && Math.floor(Math.log(value)) == Math.floor(Math.log(tv))) { return null; } } return value + ' ns'; } }, position: 'right', id: 'y-axis-1', }], } } }
原task对象的接入
我个人地理解里,在我们特别是游戏服务器的使用场景,一般是有个大的任务,里面调用很多个不同的SDK或者模块。在这种场景中,能够使外部模块和外部系统的接入能够方便地接入到我们地协程中就更加地实用。
GCC 10.1.0 的坑
符号问题
我发现在 GCC 10.1.0 中,如果多个文件可能会引用到协程库的时候,链接时会报类似这样的错误
看提示和出错的符号名感觉应该是引用到的相关头文件应该要生成弱符号,结果生成了强符号导致链接不过。这个问题仅在Windows上出现(我的测试环境是MSYS2带的MinGW64环境, GCC 10.1.0 ),Linux下正常。初步查了下Windows下输出的中间文件 .obj
中生成了代码(T)符号和弱(W)符号。而在Linux下只生成了弱(W)符号。看涉及的符号名感觉是GCC的BUG,猜测是GCC 10.1.0实现生成的符号可见性有点问题,我没有过多的深究。
生命周期和析构
同样,这个问题和导致的使用上的限制在 copp:future::task_t
上也一样。
Apple clang 9 和 Clang 6 开编译优化后访问协程栈变量崩溃问题
比如对 char buffer[256] = {0};
这段代码,Clang 6生成的代码是:
而Clang 7生成的没问题的代码是:
https://godbolt.org/e?readOnly=true&hideEditorToolbars=true#z:OYLghAFBqd5QCxAYwPYBMCmBRdBLAF1QCcAaPECAM1QDsCBlZAQwBtMQBGAFlICsupVs1qhkAUgBMAISnTSAZ0ztkBPHUqZa6AMKpWAVwC2tENwAcpLegAyeWpgByxgEaZiIAOwAGUgAdUBUJ1Wj1DEzNLAKC1OjsHZyM3Dx9FZUxVEIYCZmICMONTCzSVWNps3IJ4p1d3L18FHLyCiOLGyurE5PqASkVUA2JkDgByKQBme2RDLABqcXGddUbiTGYjBexxbwBBCamZzHnF9SMRPD9N7b3JSdppgzmFnQA3DKJiK93r/fvD450RkwRhIAE8vntvrcDo8js9MAAPPzuPBA%2BhsAD0aGIAzUDgh1xWBlUswA7sxCAB9BSoIGUtRAhTzTyyXazdmzewEWbsKgEemozAKBasnYc2aNdAgECI5HEQXo1jS7G4%2ByYSkIETodjPTazTXaHXjUXi8lUml0hlCiBc2YEHqzEA8zB8gWMiD20j6rXsCC0AysVh%2BAjEB3iFnhgAiP080eNP12RJJCgMCmR2mptPVVqZ4ZNHLN/It2cFwskADZZugRddxSm09ZM5bSxBC02S4ypJXKegHU70BAe2GI7HaxyXKh9LNmG3Vsx0KCIA7aKhEcNg8z8%2BL2ZLlbiAc95pJJCxA9PZ2sF06pJID4sqwA6Xn8nN3nQSghSmWGmts7fs1YCEGWhH2fN0hTvSNZm8X8xQ5KMEzg9kXlQPB0HPCki1TdMB13GUkRRNEciVFASFVBwNR9OFFj1BBl1XBF125PMx3/dAHwNbVqKghBYP/HdPz3AxmJo%2B8b1PVgMPNbDrGvY831mCAAFolPYsCczDUT3zw6xWD4%2BDR2%2BP9kNQ9CZ0wylVhTIEl1mFc10wDcWOM7c8LQYSFPEthJPMqkrOMDgj1vXV7zUl0X1LBSdJ/eMXPFdjOPYY4oP9QNg0%2BWKkOZaNvlHTLCRDYluRVYS1XpZgFAAa03ViP2IIrZj8HEjDwJR6VBZEarijlkE1YgACoP2YZBKspFwDCoKh3H0/8mtpVrsw6zAlyddoRrGiapuIP0AyDEMehYwysvFEq8WzCrquATB%2BUA4DKVQFw%2BHeWznOO/jbuIEDTrKnIqrzAgEFahDMv47LavFZhhNQTlaGCNhqRk7QXojbr/w%2BkCdII%2BUiLYaV6xwykHDeT4RzjLdtwQ1Hpyh2YqHseH8esZHydB9GPy/WVCK0Yi8cR9BCcwYnDrJ8GDJyyE3pMtDZgMWhEswfmHODEJmcpyXZhQ6X0cpTWB2HWQjtByGiFmUE8GUfmXjYAxlt63JBvndBQy69W6xydbxsm9xkundAnZm0HZjZzG5QVHmQEZjMiem0mA/FNX47y0Varmlq2oIJbBtThbYNq76KN%2ByqIGz9PM5hnonRL5b7H1tXgeT3Zjeh1A5WYD5ZjQSlfIIVsLOLcCy0rBF6KV5iUay4O%2BfbAe81mBEwfyo7rnz86qtp3vzSzGeKzJTD3Frly7eIWZPa28QAFZpEkc/ywvqCFnvlkYKT2q3P3EL3xvNxgHsO1oc77uN4oqCW/NqAOnczYWxPptaaIMeqoC7oWXehBYHkzfh5D%2BQVrCzFQFQDuZEzpyUkJg0%2B3tMHRTAYvcW1xbRnHsKrKEkwqBYDwZSSkyA/B%2BHYWRUqDhhQuXQSJHQh4bxsI4VwleCgiEKTEZw7hOJeEQXISA3SudbjWDwFQWqbZ%2B6vkLDHaQsxOAL3JlQDeBB965xcqSQGSVzHuA4lRI8t8d5gDAPo4gjjDSYAfOgOgy0egH3VoIzyx5mATg8ibfyQJpGYI8U%2BcKA9gEcxiizAse9PHywfNEgJAc64uTZjBKhIw%2BhKhGOfEYpBTAjG8JU1AIARjCJkHICUAwhhwluJwSpBAGm1MCaQSqZhvAPk4OMHw59xjjHPpIAAnJwW4MzyxCAadwSp1TamkHqSMSpUjfA9JqSU0gcBYAwCgKciASA0BGD8HgdgZAKAQCuTcu5KBhCiHLN4XwdNWAWOIFIiALhemVJcPTYgoIGldNIFcoiAB5WgrBwUHNIFgM4oh2BAuRXgVYmQ3hSKRYiDIwlRgbK5MoCFlTWB4BcMQXIoI9BYHJaQEMqJyV9BoPQJgbAOA8H4IIN5Yg5ByCEFSqRkA%2BgtzKHipSMLbxKQAOreVmPKxEIZmBKqoJIpSBAmRKUlA/CQzSZCSDVbqz8lKXAP3NQa2QMgdnpEyBoCA1gWimE4FYbQnRageDddEYIdAXWCF9WUT1SQ6huqUKULITR8j6EKIICN7wo0dDVF0MNiho0BvDdGkN3ROB9BpIMYYXBSkNIqVUjFWyETmHLEpcs3AO78tmOWB8wzvCKVwIQEgR5xhutmHoa5tyyGdIdE0m10hulAv6YM8%2BvgymrNIEYEA5ZOAjO4J4Tg1aeDrs3efctSKtk7JAHsydRzzmXNpM89w5BKBPMHR4aYIhgA%2BC%2Bbc35/zAVIpBbQWljLoXczhQijFKLH3oqRfgbFahcUYoJcgIlGLSVlMheamlYL6WjEhcyxdIwulsroIwU83LeACEkEIR9KBBUyGFS4UVS5NnKzoFKmVSqFVnmVQiVV6rNXaqVXq8YkZrVyGNTxs1VLLVUoE7akoibHXOtjREN11gc1pqDSETN/hAh%2BtoEp71UmHXlAzXJ11umygVDyNp%2BNBnwhGfaGZlNXri39ELdykt5S1kVoaVWmtdaG2PtmJ4FtLb234HbhMXt/bL3HwmJIEdFHx1MsnX0BAl46i0enbOlZbn90NMPceg5/SymSEqYu5dq7t3LrXRu5du71l1Oy/FvLp74DHMQCAXEfhhLXseReu9ggFadp0%2By/DXKuC8FJDSvwrLlmub3RsrZoWySEAQHPattb60PtEE2gLGz9l9L6Glqb87F3jHMA%2Bcw0zzCjLO5IC7x3qvue2YoI99W%2BmNdgEgNrHWHm3peSRvrHxBCDc5einlY3mATew4cspZaaubIaVFhbANltebW42/zraJ15cS8ljwqWQAzoO5l2bdXdnPZKS5wrC6QDHdO%2Bdy7tPbszdqw9nbZOBl45mad8sszuDljrT4cwng61TfGITpnGPdtTYpzDg9pP%2BnEyCBobgQA%3D%3D%3D
写在最后
Last updated
Was this helpful?