Thursday, October 30, 2008

Django on Glassfish via Jython

Introduction


For the last major web application that I created, I used Django, and absolutely loved it. Since I'm now working on a project that uses Glassfish, I decided to make an attempt at running Django in Glassfish. Since Glassfish is primarly Java-based, it makes the most sense to get Django running on Jython, and run it in Glassfish that way.

Luckily for me, some people have already done considerable work in getting Django running in a Jython environment, and on Glassfish. The information from Frank Wierzbicki's blog gave me a good sense of the state of this task. His instructions for running Django in Jython, and then running the combination in Glassfish can be found here:

Jython and Django Progress Part I
Jython and Django Progress Part II

Upon further investigation, it seems that in the months following Frank's blog posts, some other users have created a much neater way of getting everything running. In fact, getting Django running on Jython should now be as simple as following the steps listed here:

Django on Jython

And then deploying the Jython+Django on Glassfish has also been simplified:

Django .war Deployment

So, I began by following these instructions. However, along the way, I ran into a few speedbumps. In case others have a similar configuration to me, and run into the same problems, it seems prudent to document my experience here.

I'm going to post a step-by-step account of the process that I followed. I want to make it very clear that a large portion of this process involved following the steps in the above links, so much of the credit should go to those authors. Consider this an incremental contribution.

Environment


For purposes of repeatability, here is the environment in which I'm working:

  • Ubuntu Server 8.04 LTS

  • Java 1.5.0_15-b04

  • Jython SVN r5530

  • Django SVN r9294

  • Django-Jython SVN r33

  • Glassfish v2ur2-b04

  • Modjy_0_22_2



It sure is hard not to type Djython!

Process



  1. Install jython-dev

    svn co https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython/ jython-dev
    cd jython-dev
    ant


  2. You also want to add related environment variables and aliases to your environment. You can either run these each time at the command line, or add them to the appropriate shell login script (I used .bash_profile). I used the absolute path the the SVN version of jython that I'd just built.

    $JAVA_HOME = /path/to/java
    $JYTHON = /path/to/jython
    alias jython25=/path/to/jython-dev/dist/bin/jython



  3. Install Django

    svn co http://code.djangoproject.com/svn/django/trunk/ django-dev
    cd django-dev
    jython25 setup.py install

    You might also want to add a shortcut to django-admin through jython:

    alias django-admin-jy="jython25 /path/to/jython-dev/dist/bin/django-admin.py"



  4. Get the Django-Jython helper scripts

    svn co http://django-jython.googlecode.com/svn/trunk/ django-jython
    cd django-jython
    jython25 setup.py install



  5. Create a new project

    django-admin-jy startproject myproject



  6. Try to run the project:

    jython25 myproject/manage.py runserver


    It was at this point I encountered my first error:

    Traceback (most recent call last):
    File "myproject/manage.py", line 4, in
    import settings # Assumed to be in the same directory.
    java.lang.ArrayIndexOutOfBoundsException: 6696


    I remedied this error by removing the first line of myproject/settings.py -- I'm not sure why this caused a problem, and I haven't spent much more time investigating it. It doesn't seem consistently repeatable.



Now, I successfully had my Django app running in Jython. All in all, those instructions were pretty good! Next, I followed the procedure to deploy the Django/Jython app as a war.


  1. Add 'doj' to the app list in settings.py. In case you're wondering, this was added to your jython site-packages when you installed django-jython.

  2. Build the war using the manage.py module built from django-jython:

    jython25 myproject/manage.py war

    Wow, now you have war! It's that simple! Well, not quite, for me.


  3. Deploy the war:
    asadmin deploy myproject.war

    Depending on your Glassfish configuration, you may need to use the absolute path for the asadmin program.

    This is where my real problems began


  4. Problem:
    WARNING: com.sun.enterprise.deployment.backend.IASDeploymentException: PWC1651: Class com.xhaus.modjy.ModjyJServlet has unsupported major or minor version numbers, which are greater than those found in the Java Runtime Environment version 1.5.0_15

    That one is easy. The modjy included in django-jython was compiled with Java 1.6. I'm running Java 1.5. So, I downloaded Modjy 0.22_2 and re-built it with my version of Java. The resulting jar is copied to:django-jython/doj/management/commands/war_skel/WEB-INF/lib. Then, rebuild django-jython, and then rebuild your war.


  5. Problem:
    Command deploy executed successfully with following warning messages: Error occurred during application loading phase. The application will not run properly. Please fix your application and redeploy.
    WARNING: com.sun.enterprise.deployment.backend.IASDeploymentException: ContainerBase.addChild: start: LifecycleException: javax.servlet.ServletException: Unable to import 'modjy_servlet' from /var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar: do you maybe need to set the 'modjy_jar.location' parameter?


    This was tricky to debug, because the actual Python exception was being caught and re-raised, hiding the original cause of the error. The exception was being thrown when the xhaus Java servlet tried to import the Python code (grepping for strings helped me to determine this). Since I really wanted to see the Python error, I made the following modification to the Java part of the modjy component:

    StringWriter sw = new StringWriter();
    PrintWriter pw = new PrintWriter(sw);
    ix.printStackTrace(pw);

    throw new ServletException(sw.toString()+"--Unable to import '"+MODJY_PYTHON_CLASSNAME+"' from "+modjyJarLocation+
    ": do you maybe need to set the 'modjy_jar.location' parameter?");}


    After rebuilding modjy as described above, and redeploying, I was able to see the real cause of the problem.

  6. WARNING: com.sun.enterprise.deployment.backend.IASDeploymentException: ContainerBase.addChild: start: LifecycleException: javax.servlet.ServletException: Traceback (most recent call last):
    File "", line 1, in
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 24, in
    ImportError: No module named types


    Well.... crap. Jython was having trouble importing standard modules. According to modjy, it added to the python path every subdirectory of WEB-INFO/lib-python in the .war. But it turns out, after peeking in the code, that this is not actually the case. Modjy actually just searches for .pth files (python path files), and uses those to modify the system path. So, this was an easy remedy. For each subdirectory of lib-python that contained a library I wanted, I had to create <name>.pth, and include one line in the file, indicating the path that should be included in sys.path. I did this for django, Lib, doj, and myproject. For example: django.pth contained one line:
    /django


    This solved the problem, and the app was successfully deployed. Hooray!


  7. Browsing to localhost:8080/myproject allows me to see the deployed Django/Jython app, and results in an instant "un-hooray":
    Traceback (most recent call last):
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 80, in service
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy_exceptions$py.class", line 91, in handle
    modjy_exceptions.NoCallable: Error loading jython callable 'handler': No module named core


    This problem took a lot of poking around to debug. The problem occurs in the servlet's application.py, a small module that generates the WSGI servlet handler for Glassfish. The problem was that the system path provided seemed to include too much, causing the django import to fail. It seemed that django was resolving to a javapackage, instead of a Python module. I'm not sure exactly why this fixed it, but moving the sys.path element '__classpath__' from the middle to the end of sys.path fixed it. So, I added these lines at the top of application.py:


    import sys
    sys.path.remove('__classpath__')
    sys.path.append('__classpath__')




  8. Problem:
    Traceback (most recent call last):
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 77, in service
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 94, in dispatch_to_application
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy_wsgi$py.class", line 141, in set_wsgi_environment
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy_wsgi$py.class", line 115, in set_wsgi_streams
    TypeError: coercing to Unicode: need string, 'javainstance' type found


    This occurred when trying to store environment variables for the WSGI environment. The module used PyFile to wrap InputStreams. According to Jython documentation, streams should be wrapped using org.python.core.util.FileUtil.wrap(). So, I added
    from org.python.core.util import FileUtil
    And changed the lines

    dict["wsgi.input"] = PyFile(req.getInputStream())
    dict["wsgi.errors"] = PyFile(System.err)

    To read:

    dict["wsgi.input"] = FileUtil.wrap(req.getInputStream())
    dict["wsgi.errors"] = FileUtil.wrap(System.err)


    This fixed the problem.


  9. Next problem:
    Traceback (most recent call last):
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 81, in service
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy_exceptions$py.class", line 91, in handle
    modjy_exceptions.ApplicationException: dictionary update sequence element #0 has length 1; 2 is required


    This one was a real pain. Because of the exception handling magic in modjy, most of the stack trace was getting swallowed. After foolishly trying to scatter file-writing debugging statements all over the place, I realized I should just make the exceptions fire properly. So, around line 81 in modjy.py, I got rid of the try/except clauses around the line. This worked, and I got the full-blown stacktrace:


    Traceback (most recent call last):
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 73, in service
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 94, in dispatch_to_application
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib/modjy.jar/modjy.py", line 110, in call_application
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject//application.py", line 10, in handler
    to_return = h(environ, start_response)
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/django/django/core/handlers/wsgi.py", line 240, in __call__
    response = self.get_response(request)
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/django/django/core/handlers/base.py", line 67, in get_response
    response = middleware_method(request)
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/django/django/contrib/sessions/middleware.py", line 10, in process_request
    session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME, None)
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/django/django/core/handlers/wsgi.py", line 177, in _get_cookies
    self._cookies = http.parse_cookie(self.environ.get('HTTP_COOKIE', ''))
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/django/django/http/__init__.py", line 256, in parse_cookie
    c.load(cookie)
    File "/var/glassfish/domains/domain1/applications/j2ee-modules/myproject/WEB-INF/lib-python/Lib/Cookie.py", line 621, in load
    self.update(rawdata)
    ValueError: dictionary update sequence element #0 has length 1; 2 is required


    After much more poking around near the line in problem, it turns out the the Jython cookie library only handles regular strings, not unicode strings. My first attempt, which was to allow the Cookie.load function to parse unicode as well, was a bad idea, because a lot of downstream functions also needed strings rather than unicode. So, I added a pre-processing step in the servlet application.py file to convert anything unicode back into a string:


    def handler(environ, start_response):
    for k,v in environ.items():
    environ[k] = str(v)


    This fixed the problem, and the Jython/Django application successfully loaded via Glassfish. Hooray!

    Incidentally, jython should probably add handling to this error, similar to python (which complains if you try to load a Cookie via a unicode string).





I welcome any feedback from others who read my experience, either to call my debugging techniques primitive (and suggest better ways), or to provide your own experiences with running Django/Jython on Glassfish.

4 comments:

brosner said...

Huge thanks. This post came at the right time for me :)

Mohamed Lrhazi said...

So you did not need to install/config PostgreSQL?
Thanks.

Alan Kennedy said...

Hi,

I'm the maintainer of modjy, and I apologize for the problems you've had running it.

They've all been fixed now. I'll go through them one by one.

#6: Subdirectories of lib-python.

Initially, when Leo was developing Django-on-Jython, he adopted the convention of adding every subdirectory of lib-python to sys.path. However, this was not a clean solution, because it required that actual libraries be stored two subdirectories down, e.g. "django" would be stored in "lib-python/django/django", which was counter-intuitive.

So I adopted a different solution for modjy, which was to add only the lib-python directory to sys.path. This means that django, for example, would be stored directly under lib-python/django; no deeper subdirectories required.

The reason why Leo adopted his original convention was that ".egg", ".jar" file, etc, would be also be added to sys.path, being entries in the lib-python directory.

This was a good solution to the problem, but I thought it better to use the standard python convention of .pth files. Listing .eggs, .jars and .zips in .pth files was more explicit (remember "explicit is better than implicit"), and also standard.

The .pth solution is what is explicitly stated in the modjy documentation, so the statement "According to modjy, it added to the python path every subdirectory of WEB-INFO/lib-python in the .war. But it turns out, after peeking in the code, that this is not actually the case." The actual case is what is documented, and not otherwise.

Lastly, you should not need a single .pth file for every library or directory; you can use a single .pth file, perhaps "all.pth", which lists every path you want added to sys.path. So all.pth could look like this

#--------------
django
Lib
doj
myproject
#--------------

#7 The reason for this is the duplicated sub-directories, i.e. with the new modjy .pth convention, django no longer lives in "lib-python/django/django", as described above.

#8 This was a bug in modjy, which was the unfortunate result of me not being aware of an API change relating to constructing PyFile from InputStream. I raised it as a bug in the jython tracker, and have since fixed the bug.

http://bugs.jython.com/issue1171

Until the next beta release of jython 2.5, everyone using modjy should download the latest from SVN.

https://jython.svn.sourceforge.net/svnroot/jython/trunk/jython/extlibs/modjy_0_25_1.zip

Thanks for taking the time to write all this up; it will help people to get up-and-running with Django-on-Jython.

I just wanted to be sure that folks had up-to-date information on some of the details.

Regards,

Alan.

Leo Soto M. said...

Hi.

Re: #9, I've filled Jython
bug #1245
. In the meantime, I'm applying a variant of your workaround on django-jython.

BTW, I'm working on fixing all the problems we have now with using django-jython with the latest releases of jython and modjy. I've started a branch which should be merged soon on trunk but you can check it out at: http://django-jython.googlecode.com/svn/branches/modjy-0.25-integration