Hue errors out "Could not connect to hostname:9290"

Env:

Hue 3.6

Symptom:

Hue log errors out "Could not connect to hostname:9290".
Eg:

[22/Dec/2014 05:51:16 -0800] base         ERROR    Internal Server Error: /jobbrowser/
Traceback (most recent call last):
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/handlers/base.py", line 117, in get_response
    response = middleware_method(request, e)
  File "/opt/mapr/hue/hue-3.6.0/desktop/core/src/desktop/middleware.py", line 91, in process_exception
    response = render_json(exception.response_data)
  File "/opt/mapr/hue/hue-3.6.0/desktop/core/src/desktop/lib/django_util.py", line 290, in render_json
    json = encode_json(data, indent)
  File "/opt/mapr/hue/hue-3.6.0/desktop/core/src/desktop/lib/django_util.py", line 262, in encode_json
    return simplejson.dumps(data, indent=indent, cls=Encoder)
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/encoder.py", line 202, in encode
    chunks = list(chunks)
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/encoder.py", line 426, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/encoder.py", line 400, in _iterencode_dict
    for chunk in chunks:
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/encoder.py", line 434, in _iterencode
    o = _default(o)
  File "/opt/mapr/hue/hue-3.6.0/desktop/core/src/desktop/lib/django_util.py", line 72, in default
    return simplejson.JSONEncoder.default(self, o)
  File "/opt/mapr/hue/hue-3.6.0/build/env/lib/python2.6/site-packages/simplejson-2.0.9-py2.6-linux-x86_64.egg/simplejson/encoder.py", line 177, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: TTransportException('Could not connect to localhost:9290',) is not JSON serializable

Root Cause:

The error message implies they can not connect to jobtracker thrift plug-in.
Multiple reasons are for this issue:
1. jobtracker configuration is set wrongly in section "[[mapred_clusters]]" of /opt/mapr/hue/hue-<version>/desktop/conf/hue.ini
2. Network connection issue between Hue and jobtracker.
3. No jobtracker for this hadoop cluster, eg, Yarn only.

Solution:

For different reasons, we need to fix differently.
1. Correct section "[[mapred_clusters]]" of /opt/mapr/hue/hue-<version>/desktop/conf/hue.ini if this is MR1 hadoop cluster.
2. From Hue server, run "telnet hostname 9290" to test if the port on the jobtracker host is accessible.
Also, please check on the jobtracker host to verify if the jobtracker is started by running "jps -m|grep -i tracker"; and make sure jobtracker thrift plug-in port is occupied by the jobtracker process by running "lsof -i:9290".
For example:

[root@n3a ~]# jps -m|grep -i jobtracker
15601 JobTracker
[root@n3a ~]# lsof -i:9290
COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
java    15601 mapr  113u  IPv4 105159242      0t0  TCP *:9290 (LISTEN)

3. If this is Yarn(MR2) only hadoop cluster, we need to fix the entries in /opt/mapr/hue/hue-<version>/desktop/conf/hue.ini as below.
For section "[[mapred_clusters]]", we disable the access to jobtracker by imputing "submit_to=False";
For section "[[yarn_clusters]]",  we need to make sure all entries are correct and input "submit_to=True".