I can't download BTC/USDT from binance,could you send me the data from mail?

[domodo2012]

hi, i'm interested in basana，but when i follow your quick start,I couldn't download data from binance with the command below:

python -m basana.external.binance.tools.download_bars -c BTC/USDT -p 1m -s 2021-01-01 -e 2021-01-31 > binance_btcusdt_min.csv

the web is banned.
could you send data to me from mail,or upload the data on github,please ?

thanks

[gbeced]

binance_btcusdt_min.zip

[domodo2012]

got it, thank you! :)

[dairycow]

More on this issue, I believe it's due to the encoding the Windows cmd/powershell writes the file to.

I had to modify the encoding of the file using this script:

import os
import sys
import chardet


def convert_file_to_utf8(file_path):
    # Detect the file's encoding
    with open(file_path, 'rb') as f:
        data = f.read()
    detected_encoding = chardet.detect(data)['encoding']

    # Convert the file to UTF-8
    with open(file_path, 'r', encoding=detected_encoding, errors='replace') as f:
        data = f.read()
    with open(file_path, 'w', encoding='utf-8') as f:
        f.write(data)

    print(f'{file_path} converted from {detected_encoding} to UTF-8.')


if __name__ == '__main__':
    if len(sys.argv) != 2:
        print('Usage: python convert_to_utf8.py <file_path>')
        sys.exit(1)

    file_path = sys.argv[1]
    if not os.path.exists(file_path):
        print(f'Error: File "{file_path}" not found.')
        sys.exit(1)

    convert_file_to_utf8(file_path)

Which is run via python convert_to_utf8.py binance_btcusdt_min.csv

[dairycow]

Hey, sure sorry I do now realise my issue was different.

I also modified line 82 of samples/backtesting_bbands.py: exchange.add_bar_source(csv.BarSource(pair, "binance_btcusdt_day.csv", "1m"))

When I was in the debugger, I was getting weird characters for the row_dict. The issue was that the csv file which is saved from the first command was being decoded maybe in UTF-16, so when it was being read back in it was having issues reading it. Hence, the script that converts it fixed my issue.

(.venv) PS C:\Users\harry\Documents\repos\basana> python -m basana.external.binance.tools.download_bars -c BTC/USDT -p 1d -s 2014-01-01 -e 2021-01-31 > binance_btcusdt_day.csv
(.venv) PS C:\Users\harry\Documents\repos\basana> python samples/backtesting_bbands.py
Traceback (most recent call last):
  File "C:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 94, in <module>
    asyncio.run(main())
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 85, in main
    await event_dispatcher.run()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 207, in run
    await super().run(stop_signals=stop_signals)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 119, in run
    async with helpers.TaskGroup() as tg:
  File "C:\Users\harry\Documents\repos\basana\basana\core\helpers.py", line 42, in __aexit__
    await asyncio.gather(*self._tasks)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 188, in _dispatch_loop
    dispatched_dt = await self._dispatch_next(last_dt if self._strict_order else None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 162, in _dispatch_next
    self._prefetch()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 141, in _prefetch
    if event := source.pop():
                ^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 87, in pop
    ret = next(self._row_it)
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 61, in load_and_yield
    for ev in row_parser.parse_row(row):
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\external\common\csv\bars.py", line 39, in parse_row
    volume = Decimal(row_dict["volume"])
                     ~~~~~~~~^^^^^^^^^^
KeyError: 'volume'

[gbeced]

Hey, sure sorry I do now realise my issue was different.

I also modified line 82 of samples/backtesting_bbands.py: exchange.add_bar_source(csv.BarSource(pair, "binance_btcusdt_day.csv", "1m"))

That last parameter, "1m", should be in sync with whatever you used when you downloaded the bars.

When I was in the debugger, I was getting weird characters for the row_dict. The issue was that the csv file which is saved from the first command was being decoded maybe in UTF-16, so when it was being read back in it was having issues reading it. Hence, the script that converts it fixed my issue.

(.venv) PS C:\Users\harry\Documents\repos\basana> python -m basana.external.binance.tools.download_bars -c BTC/USDT -p 1d -s 2014-01-01 -e 2021-01-31 > binance_btcusdt_day.csv
(.venv) PS C:\Users\harry\Documents\repos\basana> python samples/backtesting_bbands.py
Traceback (most recent call last):
  File "C:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 94, in <module>
    asyncio.run(main())
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 85, in main
    await event_dispatcher.run()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 207, in run
    await super().run(stop_signals=stop_signals)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 119, in run
    async with helpers.TaskGroup() as tg:
  File "C:\Users\harry\Documents\repos\basana\basana\core\helpers.py", line 42, in __aexit__
    await asyncio.gather(*self._tasks)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 188, in _dispatch_loop
    dispatched_dt = await self._dispatch_next(last_dt if self._strict_order else None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 162, in _dispatch_next
    self._prefetch()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 141, in _prefetch
    if event := source.pop():
                ^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 87, in pop
    ret = next(self._row_it)
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 61, in load_and_yield
    for ev in row_parser.parse_row(row):
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\external\common\csv\bars.py", line 39, in parse_row
    volume = Decimal(row_dict["volume"])
                     ~~~~~~~~^^^^^^^^^^
KeyError: 'volume'

[dairycow]

exchange.add_bar_source(csv.BarSource(pair, "binance_btcusdt_day.csv", "1d"))

(.venv) PS C:\Users\harry\Documents\repos\basana> & c:/Users/harry/Documents/repos/basana/.venv/Scripts/python.exe c:/Users/harry/Documents/repos/basana/samples/backtesting_bbands.py
Traceback (most recent call last):
  File "c:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 94, in <module>
    asyncio.run(main())
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "c:\Users\harry\Documents\repos\basana\samples\backtesting_bbands.py", line 85, in main
    await event_dispatcher.run()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 207, in run
    await super().run(stop_signals=stop_signals)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 119, in run
    async with helpers.TaskGroup() as tg:
  File "C:\Users\harry\Documents\repos\basana\basana\core\helpers.py", line 42, in __aexit__
    await asyncio.gather(*self._tasks)
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 188, in _dispatch_loop
    dispatched_dt = await self._dispatch_next(last_dt if self._strict_order else None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 162, in _dispatch_next
    self._prefetch()
  File "C:\Users\harry\Documents\repos\basana\basana\core\dispatcher.py", line 141, in _prefetch
    if event := source.pop():
                ^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 87, in pop
    ret = next(self._row_it)
          ^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\core\event_sources\csv.py", line 61, in load_and_yield
    for ev in row_parser.parse_row(row):
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\harry\Documents\repos\basana\basana\external\common\csv\bars.py", line 39, in parse_row
    volume = Decimal(row_dict["volume"])
                     ~~~~~~~~^^^^^^^^^^
KeyError: 'volume'

[dairycow]

I've fixed the issue by changing the encoding of the file binance_btcusdt_day.csv so no need to worry.

[ghost]

@gbeced I'd suggest in the docs, use bitstamp for example, python -m basana.external.bitstamp.tools.download_bars -c BTC/USDT. Binance is banned for access from the states.