Caught error: id nova

fault, nova, openstack

Instances won’t spawn due to a problem with the python-urllib3, python-glanceclient, and python-requests package versions (mostly just urllib3 >= 1.11)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
2016-06-20 16:44:36.617 16602 ERROR nova.api.openstack [req-9a5d30d9-60b5-420c-b872-e671591b39d0 b37ae066412d494b87640624bc7a8488 7afb230202454f939e75e13c698b2eb4 - - -] Caught error: id
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack Traceback (most recent call last):
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return req.get_response(self.application)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return self._call_app(env, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     response = self.app(environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     content_type, body, accept)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in dispatch
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     return method(req=request, **action_args)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 83, in show
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     image = self._image_api.get(context, id)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 93, in get
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     show_deleted=show_deleted)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 309, in show
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     include_locations=include_locations)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 483, in _translate_from_glance
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     include_locations=include_locations)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 545, in _extract_attributes
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     queued = getattr(image, 'status') == 'queued'
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 491, in __getattr__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     self.get()
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 509, in get
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     new = self.manager.get(self.id)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 494, in __getattr__
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack     raise AttributeError(k)
2016-06-20 16:44:36.617 16602 TRACE nova.api.openstack AttributeError: id

Add this patch to line 76

1
2
3
4
5
6
7
8
9
10
def _image_meta_from_headers(self, headers):
    meta = {'properties': {}}
    safe_decode = encodeutils.safe_decode
    for key, value in six.iteritems(headers):
        # NOTE(flaper87): this is a compatibility fix
        # for urllib3 >= 1.11. Please, refer to this
        # bug for more info:
        # https://bugs.launchpad.net/python-glanceclient/+bug/1487645
        # https://review.openstack.org/#/c/244748/1/glanceclient/v1/images.py
        key = key.lower()

Restart your services

downgrading python-requests and python-urllib3 would also work but who knows what would also break.

https://review.openstack.org/#/c/244748/1/glanceclient/v1/images.py

Make sure not to fuck up

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
2016-06-21 11:13:13.259 5507 ERROR nova.api.openstack [req-a1969570-bb6a-43ce-b69a-c066bd53a9e0 444ce5a98472496c90dc632b5e335e4c e5e7ab304fd24c6ebee4c43193529d07 - - -] Caught error: expected an indented block (images.py, line 76)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack Traceback (most recent call last):
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return req.get_response(self.application)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return self._call_app(env, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     response = self.app(environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return resp(environ, start_response)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     content_type, body, accept)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in dispatch
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return method(req=request, **action_args)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 83, in show
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     image = self._image_api.get(context, id)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 93, in get
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     show_deleted=show_deleted)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 300, in show
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     _reraise_translated_image_exception(image_id)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 298, in show
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     image = self._client.call(context, version, 'get', image_id)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 216, in call
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     version)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 203, in _create_onetime_client
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     self.use_ssl, version)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 146, in _create_glance_client
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return glanceclient.Client(str(version), endpoint, **params)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/client.py", line 37, in Client
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     module = utils.import_versioned_module(version, 'client')
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 234, in import_versioned_module
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     return importutils.import_module(module)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 57, in import_module
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     __import__(import_str)
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/v1/__init__.py", line 16, in <module>
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     from glanceclient.v1.client import Client      # noqa
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/v1/client.py", line 19, in <module>
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     from glanceclient.v1.images import ImageManager
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 76
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack     for key, value in six.iteritems(headers):
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack       ^
2016-06-21 11:13:13.259 5507 TRACE nova.api.openstack IndentationError: expected an indented block