It depends on what version you are using where the extra_vars priority is.
However, the only intent of this variable is set in the inventory
file, you shouldn't need or want to override it elsewhere.
Here is how it is working for me, using a recent ansible checkout:
I set every host's default ssh port in the inventory on the host line as
follows:
server1:2222
Then, I override it on the command line using '-e ansible_ssh_port=22'