python - 加载CKAN数据集

Question

我对ckan有一些疑问:

如何:

1. 从网络加载 CKAN 数据集
2. 将此数据集转换为 pandas 数据框

而且我需要在ckan网站上注册才能查询数据吗？

我使用的是 Pyhton 3.6.1

编辑 2: 我尝试过以下代码:

 import urllib
url = 'http://dados.cvm.gov.br/api/action/datastore_search?resource_id=92741280-58fc-446b-b436-931faaca4fb4&q=CNPJ_FUNDO:11.286.399/0001-35'
fileobj = urllib.request.urlopen(url)
print(fileobj.read())

但是，结果是这样的:

b'{"help": "http://dados.cvm.gov.br/api/3/action/help_show?name=datastore_search", "success": true, "result": {"resource_id": "92741280-58fc-446b-b436-931faaca4fb4", "fields": [{"type": "int4", "id": "_id"}, {"type": "text", "id": "CNPJ_FUNDO"}, {"type": "timestamp", "id": "DT_COMPTC"}, {"type": "numeric", "id": "VL_TOTAL"}, {"type": "numeric", "id": "VL_QUOTA"}, {"type": "numeric", "id": "VL_PATRIM_LIQ"}, {"type": "numeric", "id": "CAPTC_DIA"}, {"type": "numeric", "id": "RESG_DIA"}, {"type": "numeric", "id": "NR_COTST"}, {"type": "int8", "id": "_full_count"}, {"type": "float4", "id": "rank"}], "q": "CNPJ_FUNDO:11.286.399/0001-35", "records": [], "_links": {"start": "/api/action/datastore_search?q=CNPJ_FUNDO%3A11.286.399%2F0001-35&resource_id=92741280-58fc-446b-b436-931faaca4fb4", "next": "/api/action/datastore_search?q=CNPJ_FUNDO%3A11.286.399%2F0001-35&offset=100&resource_id=92741280-58fc-446b-b436-931faaca4fb4"}}}'

我需要像 this image 这样的结果

Answer 1

1. load a CKAN dataset from web

您链接的网站在链接“API de Dados”中有一个 Python 示例:

import urllib
url = 'http://dados.cvm.gov.br/api/action/datastore_search?resource_id=92741280-58fc-446b-b436-931faaca4fb4&limit=5&q=title:jones'
fileobj = urllib.urlopen(url)
print fileobj.read()

2. transform this dataset into a pandas dataframe

像处理任何 JSON 数据集一样，解析它并加载到数据帧中(这里没有任何特定于 ckan 的内容):

>>> import pandas as pd
>>> import json
>>> response = json.loads(fileobj.read())
>>> pd.DataFrame(response['result']['records'])

  CAPTC_DIA          CNPJ_FUNDO            DT_COMPTC NR_COTST RESG_DIA  \
0      0.00  00.017.024/0001-53  2017-07-03T00:00:00        1     0.00   
1      0.00  00.017.024/0001-53  2017-07-04T00:00:00        1     0.00   
2      0.00  00.017.024/0001-53  2017-07-05T00:00:00        1     0.00   
3      0.00  00.017.024/0001-53  2017-07-06T00:00:00        1     0.00   
4      0.00  00.017.024/0001-53  2017-07-07T00:00:00        1     0.00   

  VL_PATRIM_LIQ         VL_QUOTA    VL_TOTAL  _id  
0    1111752.99  25.249352000000  1111831.24    1  
1    1112087.29  25.256944400000  1112268.26    2  
2    1112415.28  25.264393500000  1112716.06    3  
3    1112754.06  25.272087600000  1113165.75    4  
4    1113096.62  25.279867600000  1113293.06    5  

And i need have a register in ckan website to query the data?

您不需要在您链接的网站上注册，我无需注册即可检索数据。我更喜欢使用 requests 库:

import requests
import pandas as pd

params = params={
    'resource_id': '92741280-58fc-446b-b436-931faaca4fb4', 
    'limit': 5,
}
url = 'http://dados.cvm.gov.br/api/action/datastore_search'
r = requests.get(url, params=params).json()

df = pd.DataFrame(r['result']['records'])

看起来像 limit and offset parameters probably behave like in SQL 。您可能必须将列转换为数字/日期类型，同样，这并不是 ckan 特有的，您可以在 pandas 文档中找到有关如何执行此操作的答案。

>>> df.describe()
            _id
count  5.000000
mean   3.000000
std    1.581139
min    1.000000
25%    2.000000
50%    3.000000
75%    4.000000
max    5.000000

转换很容易:

>>> for col in ('CAPTC_DIA', 'NR_COTST', 'RESG_DIA', 'VL_PATRIM_LIQ', 'VL_QUOTA', 'VL_TOTAL'):
...    df[col] = pd.to_numeric(df[col])

>>> df['DT_COMPTC'] = pd.to_datetime(df['DT_COMPTC'])

>>> df.describe()
       CAPTC_DIA  NR_COTST  RESG_DIA  VL_PATRIM_LIQ   VL_QUOTA      VL_TOTAL  \
count        5.0       5.0       5.0   5.000000e+00   5.000000  5.000000e+00
mean         0.0       1.0       0.0   1.112421e+06  25.264529  1.112655e+06
std          0.0       0.0       0.0   5.303356e+02   0.012045  6.123444e+02
min          0.0       1.0       0.0   1.111753e+06  25.249352  1.111831e+06
25%          0.0       1.0       0.0   1.112087e+06  25.256944  1.112268e+06
50%          0.0       1.0       0.0   1.112415e+06  25.264394  1.112716e+06
75%          0.0       1.0       0.0   1.112754e+06  25.272088  1.113166e+06
max          0.0       1.0       0.0   1.113097e+06  25.279868  1.113293e+06

            _id  
count  5.000000  
mean   3.000000  
std    1.581139  
min    1.000000  
25%    2.000000  
50%    3.000000  
75%    4.000000  
max    5.000000