ansible-openstack-redhat task "create the external network" Error in listing quantum networks:

I re-instanced my vms and started from scratch.
I am still getting stuck on this step “create the external network”

I have 2 centos 6.4 nodes each with 2 nics.
I’m running the playbook from my mac.

quantum_network is being executed on the admin node when it fails.

the quantum server log contains the following:

[root@Linux-OpenStack-Admin ~]# cat /var/log/quantum/server.log
2013-08-02 11:16:09 ERROR [quantum.openstack.common.rpc.impl_qpid] Unable to connect to AMQP server: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file ‘/tmp/krb5cc_164’ not found). Sleeping 1 seconds

If i
touch /tmp/krb5cc_164

the error becomes:
2013-08-02 11:31:12 ERROR [quantum.openstack.common.rpc.impl_qpid] Unable to connect to AMQP server: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Bad format in credentials cache). Sleeping 60 seconds

Here is the debug output from the ansible playbook:

kbroughton:ansible-redhat-openstack$ ansible-playbook -i hosts site.yml --start-at-task=“create the external network” -vvv

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
<10.0.9.172> ESTABLISH CONNECTION FOR USER: root
<10.0.9.170> ESTABLISH CONNECTION FOR USER: root
<10.0.9.172> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.172’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-1375460700.04-134418691941573 && echo $HOME/.ansible/tmp/ansible-1375460700.04-134418691941573’”]
<10.0.9.170> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.170’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-1375460700.04-174703238053514 && echo $HOME/.ansible/tmp/ansible-1375460700.04-174703238053514’”]
<10.0.9.172> REMOTE_MODULE setup
<10.0.9.172> PUT /var/folders/t2/h22337c12hn279xwd4s9fk7s8_088c/T/tmpyEUqoY TO /root/.ansible/tmp/ansible-1375460700.04-134418691941573/setup
<10.0.9.170> REMOTE_MODULE setup
<10.0.9.170> PUT /var/folders/t2/h22337c12hn279xwd4s9fk7s8_088c/T/tmpx8zHjY TO /root/.ansible/tmp/ansible-1375460700.04-174703238053514/setup
<10.0.9.172> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.172’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-1375460700.04-134418691941573/setup; rm -rf /root/.ansible/tmp/ansible-1375460700.04-134418691941573/ >/dev/null 2>&1’”]
<10.0.9.170> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.170’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-1375460700.04-174703238053514/setup; rm -rf /root/.ansible/tmp/ansible-1375460700.04-174703238053514/ >/dev/null 2>&1’”]
ok: [Linux-OpenStack-Controller]
ok: [Linux-OpenStack-Admin]

PLAY [openstack_controller] ***************************************************

TASK: [create the external network] *******************************************
<10.0.9.170> ESTABLISH CONNECTION FOR USER: root
<10.0.9.170> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.170’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-1375460701.17-22216503000446 && echo $HOME/.ansible/tmp/ansible-1375460701.17-22216503000446’”]
<10.0.9.170> REMOTE_MODULE quantum_network state=present login_username=admin login_password=secret provider_network_type=local login_tenant_name=admin name=external_network router_external=true
<10.0.9.170> PUT /var/folders/t2/h22337c12hn279xwd4s9fk7s8_088c/T/tmpyEUqoY TO /root/.ansible/tmp/ansible-1375460701.17-22216503000446/quantum_network
<10.0.9.170> EXEC [‘ssh’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/tmp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘10.0.9.170’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-1375460701.17-22216503000446/quantum_network; rm -rf /root/.ansible/tmp/ansible-1375460701.17-22216503000446/ >/dev/null 2>&1’”]
failed: [Linux-OpenStack-Admin] => {“failed”: true, “item”: “”}
msg: Error in listing quantum networks: [Errno 111] Connection refused

FATAL: all hosts have already failed – aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/var/tmp/ansible/site.retry

Linux-OpenStack-Admin : ok=1 changed=0 unreachable=0 failed=1
Linux-OpenStack-Controller : ok=1 changed=0 unreachable=0 failed=0

any suggestions?

thanks
kesten

Hi Kesten,

Can you check if the activemq is running and accepting connections, also can you send the quantum.conf and nova.conf file so that we check if the mq configurations are proper in the file.

Regards,
Benno

Activemq is NOT running. In fact, it appears to not even be installed. I received no errors about it though. Based on the errors and config files,

There are a bunch of qpid files installed, and i can reproduce the error I see in quantum/server.log like so:

[root@Linux-OpenStack-Admin bin]# /usr/bin/qpid-config
Failed: ConnectionFailed: (None, “Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file ‘/tmp/krb5cc_0’ not found)”)

[root@Linux-OpenStack-Admin bin]# which activemq
/usr/bin/which: no activemq in (/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)

Attached are the requested nova and quantum .conf files.

kesten

(attachments)

nova.conf (2.07 KB)
quantum.conf (632 Bytes)

Perhaps it is realted to these filed bugs

https://bugzilla.redhat.com/show_bug.cgi?id=693840
https://bugzilla.redhat.com/show_bug.cgi?id=693840

cyrus-sasl-gssapi is installed on my system (presumably as a result of the ansible script - it was a pretty minimal system, freshly installed centos 6.4)

[root@Linux-OpenStack-Admin bin]# locate cyrus-sasl-gssapi
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/changed_by
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/checksum_data
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/checksum_type
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/command_line
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/from_repo
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/from_repo_revision
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/from_repo_timestamp
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/installed_by
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/reason
/var/lib/yum/yumdb/c/546f0e164a406b4e6842ec6c503377360deabb1d-cyrus-sasl-gssapi-2.1.23-13.el6_3.1-x86_64/releasever