error with cdsAPI

Home Page Forums C3S_422_Lot1_SMHI forum error with cdsAPI

This topic contains 1 reply, has 2 voices, and was last updated by  KKlehmet 3 months ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #4976

    paugallesisardsat
    Participant

    Dear all,

    I am trying to execute a cdsapi based downloader and it returns an error.

    2018-07-23 10:28:02,447 ERROR Message: an internal error occurred processing your request
    2018-07-23 10:28:02,447 ERROR Reason: adaptor.mars.internal

    Any idea about why this might be happening?

    Find the code below as well as the response (error)

    Regards,

    Pau

    import cdsapi
    
    c = cdsapi.Client()
    
    r = c.retrieve(
        'reanalysis-era5-single-levels',
        {
            'variable':'2m_temperature',
            'product_type':'reanalysis',
            'year':[
                '2000','2001','2002',
                '2003','2004','2005',
                '2006','2007','2008'
            ],
            'month':[
                '01','02','03',
                '04','05','06',
                '07','08','09',
                '10','11','12'
            ],
            'day':[
                '01','02','03',
                '04','05','06',
                '07','08','09',
                '10','11','12',
                '13','14','15',
                '16','17','18',
                '19','20','21',
                '22','23','24',
                '25','26','27',
                '28','29','30',
                '31'
            ],
            'time':[
                '00:00','01:00','02:00',
                '03:00','04:00','05:00',
                '06:00','07:00','08:00',
                '09:00','10:00','11:00',
                '12:00','13:00','14:00',
                '15:00','16:00','17:00',
                '18:00','19:00','20:00',
                '21:00','22:00','23:00'
            ],
            'format':'netcdf'
        })
    
    r.download('/data/pau/CDS/ERA5/2m_temperature_part1.nc')

    Response [useful pieces]

    2018-07-23 10:27:08,297 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
    2018-07-23 10:27:10,238 INFO Request is queued
    2018-07-23 10:27:13,493 INFO Request is running
    2018-07-23 10:28:02,447 INFO Request is failed
    2018-07-23 10:28:02,447 ERROR Message: an internal error occurred processing your request
    2018-07-23 10:28:02,447 ERROR Reason:  adaptor.mars.internal([{'levtype': ['sfc'], 'time': ['00:00:00', '01:00:00', '02:00:00', '03:00:00', '04:00:00', '05:00:00', '06:00:00', '07:00:00', '08:00:00', '09:00:00', '10:00:00', '11:00:00', '12:00:00', '13:00:00', '14:00:00', '15:00:00', '16:00:00', '17:00:00', '18:00:00', '19:00:00', '20:00:00', '21:00:00', '22:00:00', '23:00:00'], 'class': ['ea'], 'param': ['167'], 'format': ['netcdf'], 'number': ['all'], 'expect': ['any'], 'dataset': ['reanalysis'], 'date': ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04', '2000-01-05', '2000-01-06', '2000-01-07', '2000-01-08', '2000-01-09', '2000-01-10', '2000-01-11', '2000-01-12', 
    ...
    2008-12-31']}],{}): ("retrieve,class=ea,levtype=sfc,time=00:00:00/0
    ...
    -28/2008-12-29/2008-12-30/2008-12-31,target='/tmp/tmpretraopk.grib'", Exception('[\'/usr/local/bin/mars\', \'/cache/inputs/unknown/00f9834c-9d36-4e39-9b6b-b4ca538269ab/00f9834c-9d36-4e39-9b6b-b4ca538269ab.mars\'] returned 1 [b"mars - INFO   - 20180723.082715 - Welcome to MARS\\nmars - INFO   - 20180723.082715 - MARS_HOME=/opt/ecmwf/mars-client\\nmars - INFO   - 20180723.082715 - MARS Client version: 20180719\\nmars - INFO   - 20180723.082715 - MARS Client build stamp: 20180719214344\\nmars - INFO   - 20180723.082715 - MARS Client version: 6.22.2\\nmars - INFO   - 20180723.082715 - MIR version: 0.10.0\\nmars - INFO   - 20180723.082715 - Using ecCodes version 2.8.1\\nmars - INFO   - 20180723.082715 - Using odb_api version: 0.15.8 (file format version: 0.5)\\nmars - INFO   - 20180723.082715 - Using FDB5 version: 7.13.0\\nretrieve,class=ea,levtype=sfc,time=00:00:00/01:00:00/02:00:00/03:00:00/04
    ...
    8-12-15/2008-12-16/2008-12-17/2008-12-18/2008-12-19/2008-12-20/2008-12-21/2008-12-22/2008-12-23/2008-12-24/2008-12-25/2008-12-26/2008-12-27/2008-12-28/2008-12-29/2008-12-30/2008-12-31,target=/tmp/tmpretraopk.grib\\nmars - INFO   - 20180723.082715 - Processing request 1\\n\\nRETRIEVE,\\n    DATASET    = reanalysis,\\n    CLASS      = EA,\\n    TYPE       = AN,\\n    STREAM     = OPER,\\n    EXPVER     = 0001,\\n    REPRES     = SH,\\n    LEVTYPE    = SFC,\\n    PARAM      = 167,\\n    DATE       = 20000101/20000102/20000103/20000104/20000105/20000106/200
    ...
    2300,\\n    STEP       = 00,\\n    DOMAIN     = G,\\n    TARGET     = \'/tmp/tmpretraopk.grib\',\\n    EXPECT     = ANY\\n\\nmars - INFO   - 20180723.082715 - Requesting any number of fields (request describes 78912)\\nmars - INFO   - 20180723.082715 - Calling mars on \'marscore\', callback on 56417\\nmars - INFO   - 20180723.082715 - Callback at address 10.10.10.5, port 56417\\nmars - INFO   - 20180723.082717 - Mars client is on compute-0000.cdsprod.copernicus-climate.eu (10.10.10.5) 56417\\nmars - INFO   - 20180723.082717 - Mars server is on marscore (10.10.10.8) 44879\\nmars - INFO   - 20180723.082717 - Server task is 81 [mars]\\nmars - INFO   - 20180723.082717 - Request cost: 78,912 fields, 152.614 Gbytes online, nodes: mvr0010 mvr0011 mvr0012 mvr0013 mvr0014 mvr0015 mvr0016 [mars]\\nmars - INFO   - 20180723.082733 - Transfering 163867712256 bytes\\nmars - WARN   - 20180723.082735 - GribMemoryInput::dimensions() returning 1 (hardcoded!)\\nmars - ERROR  - 20180723.082744 - Error while writing to disk (No space left on device)\\nmars - WARN   - 20180723.082744 - Closing connection with 5241384 bytes outstanding\\nmars - ERROR  - 20180723.082745 - Mars server task finished in error\\nmars - ERROR  - 20180723.082745 - Double buffer error: Assertion failed: length == buffers_[i].length_ in run, line 298 of /home/cds/git/mars-server/eckit/src/eckit/io/DblBuffer.cc(RemoteException from Connector[marsmvr-0015:9701]) [mars]\\nmars - ERROR  - 20180723.082745 - Error code is -2\\nmars - WARN   - 20180723.082745 - Got 375862428 out of 163867712256 bytes\\nmars - WARN   - 20180723.082745 - Visiting database mars : expected 0, got 180\\nmars - ERROR  - 20180723.082745 - Request failed\\nmars - INFO   - 20180723.082745 - Request time:  wall: 30 sec  cpu: 1 sec \\nmars - INFO   - 20180723.082745 -   Read from network: 358.45 Mbyte(s) in 9 sec  [38.25 Mbyte/sec] \\nmars - INFO   - 20180723.082745 -   Processing in mars: wall: 18 sec \\nmars - INFO   - 20180723.082745 -   Visiting mars: wall: 30 sec \\nmars - INFO   - 20180723.082745 -   Writing to target file: 357.67 Mbyte(s) in 1 sec  [298.48 Mbyte/sec] \\nmars - INFO   - 20180723.082745 - Memory used: 397.41 Mbyte(s)\\nmars - ERROR  - 20180723.082745 - Some errors reported\\n"]',))
    2018-07-23 10:28:02,448 ERROR   Short traceback (most recent call last):
    2018-07-23 10:28:02,449 ERROR     File "/usr/lib/python3.4/site-packages/cdscompute/cdshandlers/services/handler.py", line 52, in handle_request
    2018-07-23 10:28:02,449 ERROR       e), '')
    2018-07-23 10:28:02,449 ERROR   cdsclient.exceptions.InternalError: adaptor.mars.internal([{'levtype': ['sfc'], 'time': ['00:00:00', '01:00:00', '02:00:00', '03:00:00', '04:00:00', '05:00:00', '06:00:00', '07:00:00', '08:00:00', '09:00:00', '10:00:00', '11:00:00', '12:00:00', '13:00:00', '14:00:00', '15:00:00', '16:00:00', '17:00:00', '18:00:00', '19:00:00', '20:00:00', '21:00:00', '22:00:00', '23:00:00'], 'class': ['ea'], 'param': ['167'], 'format': ['netcdf'], 'number': ['all'], 'expect': ['any'], 'dataset': ['reanalysis'], 'date': ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04', '2000-01-05', '2000-01-06', '2000-01-07', '2000-01-08', '2000-01-09', '2000-01-10', '2000-01-11', '2000-01-12', '2000-01-13', '2000-01-14', '2000-01-15', '2000-01-16', '2000-01-17'
    ...
    '2008-12-29', '2008-12-30', '2008-12-31']}],{}): ("retrieve,class=ea,levtype=sfc,time=00:00:00/01:00:
    ...
    /2008-12-28/2008-12-29/2008-12-30/2008-12-31,target='/tmp/tmpretraopk.grib'", Exception('[\'/usr/local/bin/mars\', \'/cache/inputs/unknown/00f9834c-9d36-4e39-9b6b-b4ca538269ab/00f9834c-9d36-4e39-9b6b-b4ca538269ab.mars\'] returned 1 [b"mars - INFO   - 20180723.082715 - Welcome to MARS\\nmars - INFO   - 20180723.082715 - MARS_HOME=/opt/ecmwf/mars-client\\nmars - INFO   - 20180723.082715 - MARS Client version: 20180719\\nmars - INFO   - 20180723.082715 - MARS Client build stamp: 20180719214344\\nmars - INFO   - 20180723.082715 - MARS Client version: 6.22.2\\nmars - INFO   - 20180723.082715 - MIR version: 0.10.0\\nmars - INFO   - 20180723.082715 - Using ecCodes version 2.8.1\\nmars - INFO   - 20180723.082715 - Using odb_api version: 0.15.8 (file format version: 0.5)\\nmars - INFO   - 20180723.082715 - Using FDB5 version: 7.13.0\\nretrieve,class=ea,levtype=sfc,time=00:00:00/01:00:00/
    ...
    008-12-05/2008-12-06/2008-12-07/2008-12-08/2008-12-09/2008-12-10/2008-12-11/2008-12-12/2008-12-13/2008-12-14/2008-12-15/2008-12-16/2008-12-17/2008-12-18/2008-12-19/2008-12-20/2008-12-21/2008-12-22/2008-12-23/2008-12-24/2008-12-25/2008-12-26/2008-12-27/2008-12-28/2008-12-29/2008-12-30/2008-12-31,target=/tmp/tmpretraopk.grib\\nmars - INFO   - 20180723.082715 - Processing request 1\\n\\nRETRIEVE,\\n    DATASET    = reanalysis,\\n    CLASS      = EA,\\n    TYPE       = AN,\\n    STREAM     = OPER,\\n    EXPVER     = 0001,\\n    REPRES     = SH,\\n    LEVTYPE    = SFC,\\n    PARAM      = 167,\\n    DATE       = 20000101/
    ...
    20081229/20081230/20081231,\\n    TIME       = 0000/0100/0200/0300/0400/0500/0600/0700/0800/0900/1000/1100/1200/1300/1400/1500/1600/1700/1800/1900/2000/2100/2200/2300,\\n    STEP       = 00,\\n    DOMAIN     = G,\\n    TARGET     = \'/tmp/tmpretraopk.grib\',\\n    EXPECT     = ANY\\n\\nmars - INFO   - 20180723.082715 - Requesting any number of fields (request describes 78912)\\nmars - INFO   - 20180723.082715 - Calling mars on \'marscore\', callback on 56417\\nmars - INFO   - 20180723.082715 - Callback at address 10.10.10.5, port 56417\\nmars - INFO   - 20180723.082717 - Mars client is on compute-0000.cdsprod.copernicus-climate.eu (10.10.10.5) 56417\\nmars - INFO   - 20180723.082717 - Mars server is on marscore (10.10.10.8) 44879\\nmars - INFO   - 20180723.082717 - Server task is 81 [mars]\\nmars - INFO   - 20180723.082717 - Request cost: 78,912 fields, 152.614 Gbytes online, nodes: mvr0010 mvr0011 mvr0012 mvr0013 mvr0014 mvr0015 mvr0016 [mars]\\nmars - INFO   - 20180723.082733 - Transfering 163867712256 bytes\\nmars - WARN   - 20180723.082735 - GribMemoryInput::dimensions() returning 1 (hardcoded!)\\nmars - ERROR  - 20180723.082744 - Error while writing to disk (No space left on device)\\nmars - WARN   - 20180723.082744 - Closing connection with 5241384 bytes outstanding\\nmars - ERROR  - 20180723.082745 - Mars server task finished in error\\nmars - ERROR  - 20180723.082745 - Double buffer error: Assertion failed: length == buffers_[i].length_ in run, line 298 of /home/cds/git/mars-server/eckit/src/eckit/io/DblBuffer.cc(RemoteException from Connector[marsmvr-0015:9701]) [mars]\\nmars - ERROR  - 20180723.082745 - Error code is -2\\nmars - WARN   - 20180723.082745 - Got 375862428 out of 163867712256 bytes\\nmars - WARN   - 20180723.082745 - Visiting database mars : expected 0, got 180\\nmars - ERROR  - 20180723.082745 - Request failed\\nmars - INFO   - 20180723.082745 - Request time:  wall: 30 sec  cpu: 1 sec \\nmars - INFO   - 20180723.082745 -   Read from network: 358.45 Mbyte(s) in 9 sec  [38.25 Mbyte/sec] \\nmars - INFO   - 20180723.082745 -   Processing in mars: wall: 18 sec \\nmars - INFO   - 20180723.082745 -   Visiting mars: wall: 30 sec \\nmars - INFO   - 20180723.082745 -   Writing to target file: 357.67 Mbyte(s) in 1 sec  [298.48 Mbyte/sec] \\nmars - INFO   - 20180723.082745 - Memory used: 397.41 Mbyte(s)\\nmars - ERROR  - 20180723.082745 - Some errors reported\\n"]',))
    Traceback (most recent call last):
      File "./era5_part1.py", line 44, in <module>
        'format':'netcdf'
      File "/home/pau/glorious/packages/mynotebooks/cds_down_env/lib/python3.5/site-packages/cdsapi/api.py", line 229, in retrieve
        result = self._api('%s/resources/%s' % (self.url, name), request)
      File "/home/pau/glorious/packages/mynotebooks/cds_down_env/lib/python3.5/site-packages/cdsapi/api.py", line 312, in _api
        raise Exception("%s. %s." % (reply['error'].get('message'), reply['error'].get('reason')))
    Exception: an internal error occurred processing your request. adaptor.mars.internal([{'levtype': ['sfc'], 'time': ['00:00:00', '01:00:00', '02:00:00', '03:00:00', '04:00:00', '05:00:00', '06:00:00', '07:00:00', '08:00:00', '09:00:00', '10:00:00', '11:00:00', '12:00:00', '13:00:00', '14:00:00', '15:00:00', '16:00:00', '17:00:00', '18:00:00', '19:00:00', '20:00:00', '21:00:00', '22:00:00', '23:00:00'], 'class': ['ea'], 'param': ['167'], 'format': ['netcdf'], 'number': ['all'], 'expect': ['any'], 'dataset': ['reanalysis'], 'date': ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04', '2000-01-05', '2000-01-06'
    ...
    '2008-12-30', '2008-12-31']}],{}): ("retrieve,class=ea,levtype=sfc,time=00:00:00/01:00:00/02:00:00/03:00:00/04:00:00/05:00:00/06:00:00/07:00:00/08:00:00/09:00:00/10:00:00/11:00:00/12:00:00/13:00:00/14:00:00/15:00:00/16:00:00/17:00:00/
    ...
    -15/2008-12-16/2008-12-17/2008-12-18/2008-12-19/2008-12-20/2008-12-21/2008-12-22/2008-12-23/2008-12-24/2008-12-25/2008-12-26/2008-12-27/2008-12-28/2008-12-29/2008-12-30/2008-12-31,target='/tmp/tmpretraopk.grib'", Exception('[\'/usr/local/bin/mars\', \'/cache/inputs/unknown/00f9834c-9d36-4e39-9b6b-b4ca538269ab/00f9834c-9d36-4e39-9b6b-b4ca538269ab.mars\'] returned 1 [b"mars - INFO   - 20180723.082715 - Welcome to MARS\\nmars - INFO   - 20180723.082715 - MARS_HOME=/opt/ecmwf/mars-client\\nmars - INFO   - 20180723.082715 - MARS Client version: 20180719\\nmars - INFO   - 20180723.082715 - MARS Client build stamp: 20180719214344\\nmars - INFO   - 20180723.082715 - MARS Client version: 6.22.2\\nmars - INFO   - 20180723.082715 - MIR version: 0.10.0\\nmars - INFO   - 20180723.082715 - Using ecCodes version 2.8.1\\nmars - INFO   - 20180723.082715 - Using odb_api version: 0.15.8 (file format version: 0.5)\\nmars - INFO   - 20180723.082715 - Using FDB5 version: 7.13.0\\nretrieve,class=ea,levtype=sfc,time=00:00:00/01:00:00/02:00:00/03:00:00/04:00:00/05:00:00/06:00:00/07:00:00/08:00:00/09:00:00/10:00:00/11:00:00/12:00:00/
    ...
    -17/2008-12-18/2008-12-19/2008-12-20/2008-12-21/2008-12-22/2008-12-23/2008-12-24/2008-12-25/2008-12-26/2008-12-27/2008-12-28/2008-12-29/2008-12-30/2008-12-31,target=/tmp/tmpretraopk.grib\\nmars - INFO   - 20180723.082715 - Processing request 1\\n\\nRETRIEVE,\\n    DATASET    = reanalysis,\\n    CLASS      = EA,\\n    TYPE       = AN,\\n    STREAM     = OPER,\\n    EXPVER     = 0001,\\n    REPRES     = SH,\\n    LEVTYPE    = SFC,\\n    PARAM      = 167,\\n    DATE       = 20000101/20000102/20000103/20000104/20000105/20000106/20000107/20000108/20000109/20000110/20000111/20000112/20000113/20000114/20000115/20000116/20000117/
    ...
    /20081211/20081212/20081213/20081214/20081215/20081216/20081217/20081218/20081219/20081220/20081221/20081222/20081223/20081224/20081225/20081226/20081227/20081228/20081229/20081230/20081231,\\n    TIME       = 0000/0100/0200/0300/0400/0500/0600/0700/0800/0900/1000/1100/1200/1300/1400/1500/1600/1700/1800/1900/2000/2100/2200/2300,\\n    STEP       = 00,\\n    DOMAIN     = G,\\n    TARGET     = \'/tmp/tmpretraopk.grib\',\\n    EXPECT     = ANY\\n\\nmars - INFO   - 20180723.082715 - Requesting any number of fields (request describes 78912)\\nmars - INFO   - 20180723.082715 - Calling mars on \'marscore\', callback on 56417\\nmars - INFO   - 20180723.082715 - Callback at address 10.10.10.5, port 56417\\nmars - INFO   - 20180723.082717 - Mars client is on compute-0000.cdsprod.copernicus-climate.eu (10.10.10.5) 56417\\nmars - INFO   - 20180723.082717 - Mars server is on marscore (10.10.10.8) 44879\\nmars - INFO   - 20180723.082717 - Server task is 81 [mars]\\nmars - INFO   - 20180723.082717 - Request cost: 78,912 fields, 152.614 Gbytes online, nodes: mvr0010 mvr0011 mvr0012 mvr0013 mvr0014 mvr0015 mvr0016 [mars]\\nmars - INFO   - 20180723.082733 - Transfering 163867712256 bytes\\nmars - WARN   - 20180723.082735 - GribMemoryInput::dimensions() returning 1 (hardcoded!)\\nmars - ERROR  - 20180723.082744 - Error while writing to disk (No space left on device)\\nmars - WARN   - 20180723.082744 - Closing connection with 5241384 bytes outstanding\\nmars - ERROR  - 20180723.082745 - Mars server task finished in error\\nmars - ERROR  - 20180723.082745 - Double buffer error: Assertion failed: length == buffers_[i].length_ in run, line 298 of /home/cds/git/mars-server/eckit/src/eckit/io/DblBuffer.cc(RemoteException from Connector[marsmvr-0015:9701]) [mars]\\nmars - ERROR  - 20180723.082745 - Error code is -2\\nmars - WARN   - 20180723.082745 - Got 375862428 out of 163867712256 bytes\\nmars - WARN   - 20180723.082745 - Visiting database mars : expected 0, got 180\\nmars - ERROR  - 20180723.082745 - Request failed\\nmars - INFO   - 20180723.082745 - Request time:  wall: 30 sec  cpu: 1 sec \\nmars - INFO   - 20180723.082745 -   Read from network: 358.45 Mbyte(s) in 9 sec  [38.25 Mbyte/sec] \\nmars - INFO   - 20180723.082745 -   Processing in mars: wall: 18 sec \\nmars - INFO   - 20180723.082745 -   Visiting mars: wall: 30 sec \\nmars - INFO   - 20180723.082745 -   Writing to target file: 357.67 Mbyte(s) in 1 sec  [298.48 Mbyte/sec] \\nmars - INFO   - 20180723.082745 - Memory used: 397.41 Mbyte(s)\\nmars - ERROR  - 20180723.082745 - Some errors reported\\n"]',)).
    #4977

    KKlehmet
    Participant

    Dear Pau,

    I have tried something silimar (see below) once and it worked – also when changing to the netCDF format. In case you still have problems, could you pose your question directly to the CDS ‘Help & Support’ ?

    download-ERA5.py # create python script for downloading
    #!/usr/bin/env python
    import cdsapi
    c = cdsapi.Client()

    Script:download-ERA5.py
    c.retrieve(“reanalysis-era5-pressure-levels”,
    {
    “variable”: “temperature”,
    “pressure_level”: “1000”,
    “product_type”: “reanalysis”,
    “date”: “2008-01-01”,
    “time”: “12:00”,
    “format”: “grib”
    },
    “download.grib”)

    Best regards
    Katharina

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.