姑苏城外一茅屋，万树梅花月满天

0%

python | 如何让网络请求变得更快

发表于 更新于 分类于 ， Valine：

这一章讨论的是顺序请求，如何变得更快。

意思是，100 个请求，不是多线程执行，也不借助协程，仅仅是顺序执行。这一章的场景多用在数据的实时更新上，如股票、加密货币订单数据等。

主要解决两个问题

  • 实时性的延迟更低
  • 数据得到有效更新

参考资料



无 time.sleep 比较


普通请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import time
import requests
import json


def main():
    start = time.time()
    for _ in range(100):
        _start = time.time()
        res = requests.get('https://www.mexc.cc/open/api/v2/market/depth', params={
            "symbol": "BTC_USDT",
            "depth": "2"
        })
        print(time.time() - _start, end=" ")
        print(
            f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
2
3
4
5
6
7
0.6507468223571777 price: 46797.66 amount:4.360348
0.5217089653015137 price: 46796.84 amount:0.107818
0.5280392169952393 price: 46796.84 amount:0.107818
0.5625147819519043 price: 46780.91 amount:1.579976
0.5895538330078125 price: 46777.97 amount:7.56166
...
发送100次请求，耗时：61.30952596664429

内部 session 请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import time
import requests
import json


def main():
    start = time.time()
    for _ in range(100):
        _start = time.time()
        with requests.session() as session:
            res = session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
                "symbol": "BTC_USDT",
                "depth": "2"
            })
            print(time.time() - _start, end=" ")
            print(
                f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
2
3
4
5
6
7
0.5665690898895264 price: 46800.02 amount:1.597351
0.5575737953186035 price: 46800.02 amount:1.597351
0.5284488201141357 price: 46800.02 amount:1.597351
0.5464150905609131 price: 46800.02 amount:1.597351
0.5936479568481445 price: 46800.02 amount:1.597351
...
发送100次请求，耗时：61.48327708244324

外部 session 请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import time
import requests
import json


def main():
    session = requests.Session()
    start = time.time()
    for _ in range(100):
        _start = time.time()
        res = session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
            "symbol": "BTC_USDT",
            "depth": "2"
        })
        print(time.time() - _start, end=" ")
        print(
            f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
2
3
4
5
6
7
0.6722450256347656 price: 46836.98 amount:0.03873
0.1389920711517334 price: 46836.98 amount:0.03873
0.13564491271972656 price: 46836.98 amount:0.03873
0.13081908226013184 price: 46836.98 amount:0.03873
0.13097906112670898 price: 46836.98 amount:0.03873
...
发送100次请求，耗时：18.916182041168213

内部 session 协程请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import time
import requests
import json
import aiohttp
import async_timeout

import asyncio


async def main():
    start_time = time.time()
    for _ in range(100):
        _start_time = time.time()
        async with aiohttp.ClientSession(trust_env=True) as session:
            with async_timeout.timeout(3):
                async with session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
                    "symbol": "BTC_USDT",
                    "depth": "2"
                }) as response:
                    if response.status == 200:
                        text = await response.text()
                        infos = json.loads(text).get("data")
                        asks = infos.get("asks")
                        print(f"{time.time() - _start_time} price:{asks[0].get('price')} amount:{asks[0].get('quantity')}")
    print(f"100次请求花费的时间: {time.time() - start_time}")


if __name__ == '__main__':
    asyncio.run(main())

输出

1
2
3
4
5
6
7
0.6727511882781982 price:46830.01 amount:2.113595
0.5799381732940674 price:46830.01 amount:2.113595
0.5304088592529297 price:46821.92 amount:0.033751
0.5672211647033691 price:46827.41 amount:1.999445
0.5117049217224121 price:46827.41 amount:1.999445
...
100次请求花费的时间: 72.18698906898499

外部 session 协程请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import time
import requests
import json
import aiohttp
import async_timeout

import asyncio


async def main():
    start_time = time.time()
    session = aiohttp.ClientSession(trust_env=True)
    for _ in range(100):
        _start_time = time.time()
        with async_timeout.timeout(3):
            async with session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
                "symbol": "BTC_USDT",
                "depth": "2"
            }) as response:
                if response.status == 200:
                    text = await response.text()
                    infos = json.loads(text).get("data")
                    asks = infos.get("asks")
                    print(f"{time.time() - _start_time} price:{asks[0].get('price')} amount:{asks[0].get('quantity')}")
    print(f"100次请求花费的时间: {time.time() - start_time}")


if __name__ == '__main__':
    asyncio.run(main())

输出

1
2
3
4
5
6
7
0.9183738231658936 price:46875.93 amount:0.093727
0.1273479461669922 price:46875.93 amount:0.093727
0.1262187957763672 price:46875.93 amount:0.093727
0.124420166015625 price:46875.93 amount:0.093727
0.11751985549926758 price:46875.93 amount:0.093727
...
100次请求花费的时间: 17.95445489883423

gather 模式下的 内部 session 请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import time
import requests
import json
import aiohttp
import async_timeout

import asyncio


async def get_mexc_depth(session):
    with async_timeout.timeout(3):
        async with session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
            "symbol": "BTC_USDT",
            "depth": "2"
        }) as response:
            if response.status == 200:
                text = await response.text()
                infos = json.loads(text).get("data")
                asks = infos.get("asks")
                return {asks[0].get('price')}, {asks[0].get('quantity')}


async def main():
    start_time = time.time()
    session = aiohttp.ClientSession(trust_env=True)
    for _ in range(100):
        _start_time = time.time()
        function = [get_mexc_depth(session)]
        result = await asyncio.gather(*function)
        print(f"{time.time() - _start_time} price:{result[0][0]} amount:{result[0][1]}")

    print(f"100次请求花费的时间: {time.time() - start_time}")


if __name__ == '__main__':
    asyncio.run(main())

输出

1
2
3
4
5
6
0.5796411037445068 price:{'46856.12'} amount:{'1.648762'}
0.11725902557373047 price:{'46856.12'} amount:{'1.648762'}
0.11715507507324219 price:{'46856.12'} amount:{'1.648491'}
0.11682677268981934 price:{'46856.12'} amount:{'1.648491'}
...
100次请求花费的时间: 14.033983945846558

综上所述， session 可以有效的解决延迟问题。


探究 sleep 的影响


睡眠 0.8 普通请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import time
import requests
import json


def main():
    start = time.time()
    for _ in range(100):
        _start = time.time()
        res = requests.get('https://www.mexc.cc/open/api/v2/market/depth', params={
            "symbol": "BTC_USDT",
            "depth": "2"
        })
        print(time.time() - _start, end=" ")
        print(
            f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
        time.sleep(0.8)
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
发送100次请求，耗时：143.27592873573303

睡眠 0.8 外部 session

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import time
import requests
import json


def main():
    session = requests.Session()
    start = time.time()
    for _ in range(100):
        _start = time.time()
        res = session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
            "symbol": "BTC_USDT",
            "depth": "2"
        })
        print(time.time() - _start, end=" ")
        print(
            f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
        time.sleep(0.8)
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
发送100次请求，耗时：102.14432096481323

header 的影响


“Connection”: “close” 有session

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import time
import requests
import json


def main():
    session = requests.Session()
    start = time.time()
    for _ in range(100):
        _start = time.time()
        res = session.get('https://www.mexc.cc/open/api/v2/market/depth', headers={"Connection": "close"}, params={
            "symbol": "BTC_USDT",
            "depth": "2"
        })
        print(time.time() - _start, end=" ")
        print(
            f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
    end = time.time()
    print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    main()

输出

1
发送100次请求，耗时：82.65215396881104

无 “Connection”: “close” 有 session

根据上面的输出为

1
发送100次请求，耗时：18.916182041168213

清除缓存


我们频繁的请求一个接口，可能会遇到缓存数据。「服务器或者 CDN 缘故」

尽管，可能会延迟几十毫秒，但是，对于我们交易来讲影响非常大的，往往这个单子都被别人抢走了，但还是会返回这个单子的数据。

所以，我们可以加入 headers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import time
import requests
import json


class t:
    def __init__(self):
        self.session = requests.session()

    def main(self):
        start = time.time()
        for _ in range(100):
            _start = time.time()
            res = self.session.get('https://www.mexc.cc/open/api/v2/market/depth', params={
                "symbol": "BTC_USDT",
                "depth": "2"
            }, headers={
                "Cache-Control": "no-cache",
                "Pragma": "no-cache"
            })
            print(time.time() - _start, end=" ")
            print(
                f'price: {json.loads(res.text).get("data").get("asks")[0].get("price")} amount:{json.loads(res.text).get("data").get("asks")[0].get("quantity")}')
        end = time.time()
        print(f'发送100次请求，耗时：{end - start}')


if __name__ == '__main__':
    t = t()
    t.main()

但是，如果 CDN 等其他因为更新不及时，还是返回过期数据，那也没办法了。

请我喝杯咖啡吧～