Ondřej Čertík

Discussion about global warming

2012-01-29T22:55:00.000-08:00

I have read the text The Truth About Greenhouse Gases by William Happer, a physicist at Princeton. I liked it, so I posted it to my Google+.I was surprised by so many emotional responses. The post also made Michael Tobis to write a blog post with his opinion.

I was not satisfied about the overall tone of the discussion. I am really just interested in factual arguments. As such,I browsed through all the arguments against William's paper from the above discussion, and chose one well formulated question, that I think represents the most important objection: "In your paper you state at several places, that doubling the CO2 concentrations will only increase the temperature by 1 C. However, it is claimed (see for example Michael's post above)that the increase will be around 2.5 C. Which number is correct and where is it coming from?" I wrote to William and askedwhether he would be willing to answer it. With his permission I am posting his answer here.

Answer: sensitivity.pdf
Images referenced in the answer: image001.png, image002.png.
Link referenced in the answer: http://www.thegwpf.org/best-of-blogs/4247-steve-mcintyre-closing-thoughts-on-best.html

When double precision is not enough

2012-01-26T10:49:00.000-08:00

I was doing some finite element (FE) calculation and I needed the sum of the lowest 7 eigenvalues of a symmetric matrix (that comes from the FE assembly) to converge to at least 1e-8 accuracy (so that I can check calculation done by some other solver of mine, that calculates the same but doesn't use FE). In reality I wanted the rounded value to 8 decimal digits to be correct, so I really needed 1e-9 accuracy (but it's ok if it is let's say 2e-9, but not ok if it is 9e-9). With my FE solver, I couldn't get it to converge more than to roughly 5e-7 no matter how hard I tried. Now what?

When doing the convergence, I take a good mesh and keep increasing "p" (the polynomial order) until it converges. For my particular problem, it is fully converged for about p=25 (the solver supports the order up to 64). Increasing "p" further will not increase the accuracy anymore, and the accuracy stays at the level 5e-7 for the sum of the lowest 7 eigenvalues. For optimal meshes, it converges at p=25, for not optimal meshes, it converges for higher "p", but in all cases, it doesn't get below 5e-7.

I know from experience, that for simpler problems, the FE solver can easily converge to 1e-10 or more using double precision. So I know it is doable, now the question is what the problem is: there
are a few possible reasons:

The FE quadrature is not accurate enough
The condition number of the matrix is high, thus LAPACK doesn't return very accurate eigenvalues
Bug in the assembly/solver (like single/double corruption in Fortran, or some other subtle bug)

When using the same solver for simpler potential, it converged nicely to 1e-10. So this suggests there is no bug in the assembly or solver itself. It is possible that the quadrature is not accurate enough, but again, if it converges for simple problem, it's probably not it. So it seems it is the ill conditioned matrix, that causes this. So I printed the residuals (that I simply calculated in Fortran using the matrix and the eigenvectors returned by LAPACK), and it only showed 1e-9. For simpler problems, it can go to 1e-14 easily. So that must be it. How do we fix it?

Obviously by making the matrix less ill conditioned, which is caused by the mesh for the problem (the ratio of the longest/shortest elements is 1e9) but for my problem I really needed such a mesh. So the other option is to increase the real number accuracy.

In Fortran all real variables are defined as real(dp), where dp is an integer defined at a single place in the project. There are several ways to define it, but it's value is 8 for gfortran and it means double precision.So I increased it to 16 (quadruple precision), recompiled. Now the whole program calculates in quadruple precision (more than 30 significant digits). I had to recompile LAPACK using the "-fdefault-real-8" gfortran option, that promotes all double precision numbers to quadruple precision, and I used the "d" versions (double precision, now promoted to quadruple) of LAPACK routines.

I rerun the calculation ---- and suddenly LAPACK residuals are around 1e-13, and the solver converges to 1e-10 easily (for the sum of the lowest 7 eigenvalues). Problem solved.

Turning my Fortran program to quadruple precision is as easy as changing one variable and recompiling. Turning LAPACK to quadruple precision is easy with a single gfortran flag (LAPACK uses the old f77 syntax for double precision, if it used real(dp), then I would simply change it as for my program). The whole calculation got at least 10x slower with quadruple. The reason is that gfortran runtime uses the libquadmath library, that simulates quadruple precision (as current CPUs only support double precision natively).

I actually discovered a few bugs in my program (typically some constants in older code didn't use the "dp" syntax, but had the double precision hardwired). Fortran warns about all such cases, when the real variables have incompatible precision.

It is amazing how easy it is to work with different precision in Fortran (literally just one change and recompile). How could this be done with C++? This wikipedia page suggests, that "long double" is only 80bit in most cases (quadruple is 128bit), but gcc offers __float128, so it seems I would have to manually change all "double" to "__float128" in the whole C++ program (this could be done with a single "sed" command).

Google Code vs GitHub for hosting opensource projects

2010-11-18T18:10:00.000-08:00

Cython is now considering options where to move the main (mercurial) repository, and Robert Bradshaw (one of the main Cython developers) has asked me about my experience with regards to Google Code and GitHub, since we use both with SymPy.

Google Code is older, and it was the first service that provided free (virtually unlimited) number of projects that you could easily and immediately setup. At that time (4 years ago?) that was something unheard of. However, the GitHub guys in the meantime not only made this available too, but also implemented features, that (as far as I know) no one offers at all, in particular hosting your own pages at your own domain (but at GitHub's servers, some examples are sympy.org and docs.sympy.org), commenting on git branches and pull requests before the code gets merged in (I am 100% convinced that this is the right approach, as opposed to comment on the code after it gets in), allow to easily fork the repository and it has simply more social features, that the Google Code doesn't have.

I believe that managing an opensource project is mainly a social activity, and GitHub's social features really make so many things easier. From this point of view, GitHub is clearly the best choice today.

I think there is only one (but potentially big) problem with GitHub, that its issue tracker is very bad, compared to the Google Code one. For that reason (and also because we already use it), we keep our issues at Google Code with SymPy.

The above are the main things to consider. Now there are some little things to keep in mind, that I will briefly touch below: Google Code doesn't support git and blocks access from Cuba and other countries, when you want to change the front page, you need to be an admin, while at GitHub I simply add push access to all sympy developers, so anyone just pushes a patch to this repository: https://github.com/sympy/sympy.github.com, and it automatically appears on our front page (sympy.org), with Google Code we had to write long pages (in our docs) about how to send patches, with GitHub we just say, send us a pull request, and point to: http://help.github.com/pull-requests/. In other words, GitHub takes care of teaching people how to use git and figure out how to send patches, and we can concentrate on reviewing the patches and pushing them in.

Wikipages at github are maintained in git, and they provide the webfrontend to it as opensource, so there is no vendor lock-in. Anyone with github account can modify our wiki pages, while the Google Code pages can only be modified by people that I add to the Google Code project, which forced us to install mediawiki on my linode server (hosted at linode.com, which by the way is an excellent VPS hosting service, that I have been using for couple of years already and I can fully recommend it), and I had to manage it all the time, and now we are moving our pages to the github wiki, so that I have one less thing to worry about.

So as you can see, I, as admin, have less things to worry about, as github manages everything for me now, while with Google Code, I had to manage lots of things on my linodes.

One other thing to consider is that GitHub is only for git, but they also provide svn and hg access (both push and pull, they translate the repository automatically between git and svn/hg), I never really used it much, so I don't know how stable this is. As I wrote before, I think that git is the best tool now for maintaining a project, and I think that github is now the best choice to host it (except the issue tracker, where Google Code is better).

git has won

2010-10-31T00:26:00.000-07:00

I switched to git from mercurial about two years ago. See here why I switched and here my experience after 4 months. Back then I was unsure, whether git will win, but I thought it has a bigger momentum. Well, I think that now it's quite clear that git has already won. Pretty much everybody that I collaborate with is using git now.

I use github everyday, and now thanks to github pull requests, I think it's the best collaboration platform out there (compared to Google Code, Sourceforge, Bitbucket or Launchpad).

I think it's partly because the github guys have a clear vision of what has to be done in order to make collaboration more easier and they do it, but more importantly that git branches is the way to go, as well as other git features, that are "right" from the beginning (branches, interactive rebase, and so on), while other VCS like bzr and mercurial simply either don't have them, or are getting them, but it's hard to get used to it (for example mercurial uses the "mercurial queues", and I think that is the totally wrong approach to things).

Anyway, this is just my own personal opinion. I'll be happy to discuss it in the comments, if you disagree.

Week Aug 9 - 13

2010-08-13T20:46:00.000-07:00

On Monday I learned fwrap (excellent piece of software btw), there were a few minor technical issues, that I communicated with Kurt on the fwrap mailinglist and I also send him a simple patch, so that it works fine for my fortran code (functions returning the value itself, instead of a tuple of length 1).

Then I took my old fortran based shooting method solvers that I wrote couple years ago and wrapped them using fwrap and run couple simulations against my FE solver.

On Tuesday we had a lunch with all the advisors and students and the llnl director and a little presentation about what we did.

On Wednesday I run shooting method calculations for 50 states of silver, both for selfconsistent DFT potential and Z/r potential. I then also run the FE solver for the same DFT potential and compared results. There are lots of small technical issues, for example I had to use cubic splines to interpolate the potential, play with the mesh for the shooting method and so on.

However, the shooting method and FE agrees to every single printed digit, after making sure that the mesh is ok for both methods. For all potentials that I tried. That's very cool.

In the process of it, I also wrote a patch to SymPy to calculate exact energies for the Hydrogen atom, both from Schroedinger and Dirac equations. I still need to polish it a bit.

On Thursday I run couple more calculations and setup a poster and had a poster session, it was two hours, and I think around 7 people (not counting other students and people from our group) stopped by and talked with me about it, so I was very happy. Being able to solve radial Schroedinger and especially Dirac equations robustly is something that several people in the lab would really need.

Today I talked little bit (finally) about some Green functions in QM and QFT with a postdoc in the Quantum Simulations group, that I always wanted to, but didn't have time before, then packed my things and went back to Reno.

My plan for the next week(s) is to wrap up what I did and put it into articles. I already have enough material for some articles, so it has to be done. In parallel, I'd like to finish the FE Dirac solver, the coding is done, but now I need to play with adaptivity and also investigate if we are getting the spurious states, that other people are getting when using b-splines.

Week Aug 2 - 6

2010-08-06T18:02:00.000-07:00

This week I essentially only worked on my LLNL poster, which I finally finished about two hours ago. I have created a web page for it:

http://certik.github.com/ccms-10-poster/

where you can download pdf, sources, I also put there some relevant info and links.

It turned out to be a lot more work than I expected (well, as usual), but we were very thorough with John and in the process I discovered several bugs in my program, so I am glad we did it. I used to generate all the plots by hand, by manually adjusting all the parameters in the Python script (like atomic number, mesh parameters, element orders, adaptivity parameters, error tolerance and so on). Essentially I had to remember all these parameters for each of the plots (about 10 of them). Then I settled to have a Python dictionary, that holds all the parameters, and then I just pass them to a radial_schroedinger_equation_adapt(params, error_tol=1e-8) function.

Here are example of the parameters:


    params_hydrogen_p_L = dict(l=0, Z=1, a=0, b=100, el_num=4, el_order=1,
            eig_num=3, mesh_uniform=False, mesh_par1=20, adapt_type="p",
            eqn_type="R")
    params_hydrogen_p_U = dict(l=0, Z=1, a=0, b=100, el_num=4, el_order=2,
            eig_num=3, mesh_uniform=True, adapt_type="p", eqn_type="R")
    params_hydrogen_hp_U = dict(l=0, Z=1, a=0, b=100, el_num=4, el_order=2,
            eig_num=3, mesh_uniform=True, adapt_type="hp", eqn_type="R")
    params_hydrogen_h_U = dict(l=0, Z=1, a=0, b=100, el_num=4, el_order=6,
            eig_num=3, mesh_uniform=True, adapt_type="romanowski",
            eqn_type="rR")

    params_silver_p_L = dict(l=0, Z=47, a=0, b=150, el_num=4, el_order=13,
            eig_num=50, mesh_uniform=False, mesh_par1=35, adapt_type="p",
            eqn_type="R")
    params_silver_hp_L = dict(l=0, Z=47, a=0, b=150, el_num=4, el_order=13,
            eig_num=50, mesh_uniform=False, mesh_par1=35, adapt_type="hp",
            eqn_type="R")

I mean, this is kind of obvious if you think about it, but for some reason I didn't do that at all at the beginning, because I thought --- I'll just run this once and I am done with it. But I had to run it like 20x, e.g. regenerating he plots, then creating a table about meshes, then redoing the table after changing the error tolerance, and so on.

Besides that I also got permission to release my code, so I'll go over it in the coming days and generate nice patches against Hermes1D.

Also in the process of creating the poster, I played a lot with p-FEM, uniform-p-FEM, hp-FEM and h-FEM and I will keep playing with that. It's clear to me now, that our current Hermes1D is not optimal. Especially the convergence of hp-FEM and p-FEM (as it is implemented right now) greatly depends on the initial mesh.

Nevertheless, even with the above limitations, hp-FEM seems to be really good if you don't a-priori know anything about the problem/mesh. One should not make any deep conclusions in 1D (it might be a bit different in 2D and 3D, and also I only did couple test problems), but from my experience so far, hp-FEM is a really good choice, if you just want to solve the problem and get a decent convergence (way better than h-FEM, and in general about the same as uniform-p-FEM with optimized mesh).

Another conclusion is that uniform-p-FEM (also called spectral element method), if you optimize the mesh for the problem, is very fast. All you have to do is increase the polynomial order and it goes very straight on the convergence graphs, it's very hard to beat. Also, and that I would like to write in the coming days, the algorithm for optimizing the mesh is really simple: just solve it with high "p", then play with the mesh parameters (for logarithmic mesh, there are only 2 parameters --- number of elements, and a ratio of first vs. last element), so that the eigenvalues (that one is interested in) are converged (with given accuracy) and optimize it wrt DOFs. The algorithm can also "look" at the convergence graphs and make sure it's steep enough. For atomic problems, my experience shows that the logarithmic mesh is good enough (as long as you optimize it). The advantage is that you do this once, and then (for close enough potentials in the Schroedinger equation), you just increase "p", and it's very robust and fast (no need for reference mesh, or trial refinements and so on).

When I get back to Reno, we'll do more research on hp-FEM with Pavel and I think this is not the last word to say. We need to review how we choose the candidates for eigenvectors, especially "p" vs "hp" and make it more robust. We'll see.

Week July 26 - 30

2010-07-31T00:39:00.000-07:00

This week I have been wrapping up my work at LLNL and trying to generate some comparisons between different approaches to adapt to multiple eigenvectors at once.

It turns out that most of the issues are in the way we create the new mesh for the next adaptivity iteration, in particular, how do we choose which candidates to refine. I have tried to converge to the lowest eigenvector (as well as to any other eigenvector too), to the sum of the eigenvectors, to each eigenvector individually and taking the union of the meshes and so on. I have also implemented the uniform p-FEM as well as tried p-FEM and hp-FEM using the approaches I mentioned above.

It seems to be crucial to have a good initial mesh, at least for p-FEM. If I use a good mesh and p-adaptivity, I am able to get the best results so far. In principle hp adaptivity should be at least as good, but our current approach doesn't show it yet. Hopefully we'll manage to make it work.

Besides that I have also implemented H1 norms for the Function class (based on Fekete points) by calculating coefficients with regards to a FE basis and some other little things.

Week July 19 - 23

2010-07-23T19:06:00.000-07:00

This week I have learned how projections work in detail and wrote it up here:

http://theoretical-physics.net/dev/src/math/la.html

including all proofs (that orthogonal projection finds the closest vector and so on). At the end of the notes I have calculated some examples in 1D, so that one can see that indeed it doesn't depend on the basis and that the basis doesn't even have to be orthogonal. Then one has to use this approach to calculate the coefficients:

http://theoretical-physics.net/dev/src/math/la.html#nonorthogonal-basis

I have then implemented it in h1d. Due to the lack of time, I am now developing everything in my private branch. I'll obtain the permission in about 2 or 3 weeks, so then I'll push it into the master.

Besides that I have implemented Chebyshev points for orders greater than 48, for which I don't have exact Fekete points anymore (it'd be just a matter of running my sympy script longer, but I was hitting some accuracy issues when solving those large polynomials numerically -- one needs to obtain all roots, so Chebyshev points are ok for now). So I can now represent arbitrary polynomials in 1D.

I have implemented powering of the discrete function, it automatically determines which polynomial order it has to use and creates a new discrete function (the power) on the new mesh. I wrote lots of tests for that, and I hit an interesting bug, that my naive comparison code:


assert abs(x-y) < eps

was not good enough anymore for larger numbers and I had to read some documentation, and implement the following function:


@vectorize(0, 1)
def feq(a, b, max_relative_error=1e-12, max_absolute_error=1e-12):
    a = float(a)
    b = float(b)
    # if the numbers are close enough (absolutely), then they are equal
    if abs(a-b) < max_absolute_error:
        return True
    # if not, they can still be equal if their relative error is small
    if abs(b) > abs(a):
        relative_error = abs((a-b)/b)
    else:
        relative_error = abs((a-b)/a)
    return relative_error <= max_relative_error

Then I implemented the global H1 and L2 projections, so far the projected function is hardwired, I still need to allow the user to specify any discrete function to be projected. I need to precalculate it and so on.

I wrote bunch of tests for the projections and powers and I always discovered some bugs by writing more tests, so the progress is slow, but at least I can trust the code that is tested.

I also helped Pavel to fix couple segfaults, as well as some other things.

Theoretical Physics Reference Book

2010-07-19T22:57:00.000-07:00

Today I fulfilled my old dream --- I just created my first book! Here is how it looks like:

More images
here.

Here is the source code of the book: http://github.com/certik/theoretical-physics, the repository contains a branch 'master' with the code and 'gh-pages' with the generated html pages, that are hosted at github, at the url theoretical-physics.net.

Then I published the book at Lulu: http://www.lulu.com/product/hardcover/theoretical-physics-reference/11612144, I wanted a hardcover book, so I setup a project at Lulu, used some Lulu templates for the cover and that was it. Lulu's price for the book is $19.20 (166 black & white pages, hardcover), then I can set my own price and the rest of the money probably goes to me. I set the price to $20, because Lulu has free shipping for orders $20 or more. You can also download the pdf (for free) at the above link (or just use my git repository). So far this didn't cost me anything.

I have then ordered the book myself (just like anybody else would, at the above address) and it arrived today. It's a regular hardcover book. Beautiful, you can browse the pictures above. It smells deliciously (that you have to believe me). And all that it cost me was $19.20.

As for the contents itself, you can browse it online at theoretical-physics.net, essentially it's most of my physics notes, that I collected over the years. I'd like to treat books like software --- release early release often. This is my first release and I would call it beta. The main purpose of it was to see if everything goes through, how long it takes (the date inside the book is July 4, 2010, I created and ordered it on July 5, got the physical book on July 19) and what the quality is (excellent). I also wanted to see how the pages correspond to the pdf (you can see for yourself on the photos, click on the picasa link above).

Now I need to improve the first pages a bit, as well as the last pages, improve the index, write some foreword and so on. I also need to think how to better organize the contents itself and generally improve it. I also need to figure out some versioning scheme, so far this is version 0.1. I think I'll do edition 1, edition 2, edition 3, and so on. And whenever I feel that I have added enough new content, I'll just publish it as a new edition. So if you want to buy it, I suggest to wait for my 1.0 version, that will have the mentioned improvements.

It'd be also cool to have all the editions online somehow and create nice webpages for it (currently theoretical-physics.net points directly to the book html itself).

So far the book is just text. I still need to figure out how to handle pictures and also whether or not to use program examples (in Python, using sympy, scipy, etc.). So far I am inclining not to put there any program codes, as then I don't need to maintain them.

Overall I am very pleased with the quality, up to some minor issues that I mentioned above, everything else end up just fine. I think we have come a long way from the discovery of the printing press. Anybody can now create a book for free, and if you want to hold the hardcopy in your hands, it costs around $20. You don't need to order certain amounts of books, nor partner with some publisher etc. I think that's just awesome.

Week June 12 -- July 16

2010-07-16T22:06:00.000-07:00

This week I was implementing hp-adaptivity based on H1 projections first for the ground state and later for other states.

It took me the whole week to debug things and I am still not done. There were issues with normalizing vectors and other problems. I am now able to converge any individual vector and that seems to work fine, but I am still not able to converge multiple vectors somehow.

I spent also some time with speeding up my implementation of fekete points, and I have implemented the fast evaluation based on Lagrange interpolation polynomials. In general, I made for example the l2_norm() method about 10x faster. I think it's about as fast now as if it was written in C++ directly. The main loops are now optimized C without any Python C/API calls.

My plan is now to implement taking squares of the solutions (exactly), and so on, and projecting it back to the original mesh. I'll try converging to that. At the same time, I'll try to converge to multiple eigenvalues. Hermes1d needs some improvements, that take me quite some time to implement.

Week June 5 -- July 9

2010-07-10T12:48:00.000-07:00

This week I have learned how to use the new C++ support in Cython, and posted a demo project at github (http://github.com/certik/cpp_test), then I have refactored the Python wrappers in hermes1d and I think things are now way cleaner.

Then I have finished my reimplementation of Romanowski's algorithm for adapting eigenvalues for the radial Schroedinger equation, I've extracted his values from the graphs, calculated the same thing myself and it agrees perfectly.

I am now polishing the h1 adaptivity in h1d, I essentially reimplemented it myself, so that I can consider more candidates (all possible combinations of "p" and "h", and also soon I'll consider bisecting, trisecting and so on of the interval).

I've implemented hydrogen wavefunctions in sympy and use it to project onto a very fine mesh (12 order, lots of elements), convert to Fekete points and use it for the adaptivity. It's in pure Python, as I need to develop very fast to get some results and see what approach is the best. Everything is fast, except the selection of candidates, which I'll now rewrite into Cython and try to use hermes1d whenever possible.

Next week I'll try to merge my adaptivity with hermes1d adaptivity and make it fast. And see how it converges.

I've also spent about 10 hours with improving Pavel's hermes2d branch, as well as implementing the Vector class in hermes_common.

Week June 28 -- July 2

2010-07-03T01:41:00.000-07:00

On Saturday I flew to Prague, then to Pilsen to the ESCO2010 conference. I had to presentations there and I met lots of awesome people.

I discussed with Dmitri how to solve the Euler equations using FEM and I think I know how to implement it now in hermes2d. I will try to give it a shot at the end of the summer, as this is something that was bothering me a lot.

I have made a progress in my fekete points code, I also wrote analytic solutions for the hydrogen atom in sympy. I'll wrap it up over the weekend and push to h1d.

I have investigated how to use github for collaboration using git, here is how it looks like for sympy: http://github.com/sympy.

I have figured out how to host any domain at github, in particular this:

http://theoretical-physics.net/

now runs at github. I think this is really awesome.

Besides that, I spent some time discussing how to do hp-adaptivity in h1d with Pavel and other people, as well as I helped couple people with hermes2d and so on.

My plan for the next week is to implement hp-adaptivity in h1d for eigenproblems, as well as finish my fekete machinery and make it fast using cython and use it in conjunction with the h1d adaptivity.

Week June 20 -- June 25

2010-06-25T22:49:00.000-07:00

This week I implemented a Dirac solver in 1D, I have it in my private branch and I'll get a permission at the end of the summer to get it opensourced. Originally it didn't work at all, but then I've found a bug and it seems to be working now.

Before it I've implemented an example for the y''+k^2*y=0 equation in hermes1d by writing it as two coupled first order equations and solving the eigenproblem for "k".

I also started to implement a Function class that represents a function in 1D using Fekete points and I plan to implement orthogonal projections and adaptive algorithms for refining the mesh, so that I can play with adaptivity in 1D.

Besides that I studied couple articles about using b-splines to solve the radial Dirac equation and also about using variational formulation for FEM.

I helped fix hermes_common and did lots of administration at the lab (10 online courses and other things). We also got an awesome NIF tour.

Hermes1D patches:


Ondrej Certik (26):
      Remove the autogenerated Makefile
      hermes_common updated
      Dirac solver added
      Plot only 10 eigenfunctions
      Fix the number of equations bug
      Generate the Gauss Lobatto points
      Add the autogenerated file
      Don't use strings as dict keys
      Fekete points manipulation implemented
      fekete: solve the coefficients, tests pass
      More tests added
      system_sin_eigen example added
      Add a comment about the Lagrange interpolation
      Function.project_onto() implemented
      Function.plot() implemented
      Comparisons implemented
      Fix precision problems
      Improve tests
      Start implementing the adaptivity
      fekete: add a debug code
      Removing the Dirac solver for now
      Use smaller basis, while still getting reasonable results
      Add harmonic oscillator option
      hermes_common updated
      Use jsplot if it is available
      Better demo

hermes_common patches:


Ondrej Certik (6):
      Revert "fixed pxd"
      Update the generated .cpp file
      Increase the precision in tests to 10^-10
      Add tests for solve_linear_system_dense_lu()
      Enable scipy tests
      Test for solve_linear_system_cg() added

June 12 -- June 18

2010-06-18T21:04:00.000-07:00

This week I have implemented a JSPlot library, which allows to use matplotlib API, but plot into your web browser:
http://github.com/certik/jsplot
(Scroll down a little bit to see the screenshots and examples.)

This was the last missing piece to be able to develop with FEMhub (I don't have a root access to my computer, running RHEL5). Now I have the full Python stack working (scipy, all the solvers, mayavi, ...), plus nice plotting in the browser.

I have spent several days investigating how to write weak forms for the radial Dirac equation as well as how to derive the action for it. It turns out, that one takes the Lagrangian from quantum electrodynamics (QED), converts it to spherical coordinates (which you can imagine is a hell of a job) and then integrates over angles and what remains is the Lagrangian (resp. action) for the radial Dirac equation. People used that in the literature a lot, but I just could not figure out where they take the functional from. So that part is clear and then I went to coding.

Here are the commits for JSPlot:


Ondrej Certik (38):
      Initial commit
      Make index.html work
      README added
      Hook up raphael
      Serve raphael-min.js locally
      Plot simpler stuff
      Works fine
      plots work
      Download all the raphael stuff locally
      raphael.html added
      Use {% url %}
      Use simpler urls
      Typo fix
      Flotr demo added
      Use flotr in the index page
      Don't recalculate
      Add data by hand
      Generate the data in Python
      Add another flotr demo
      Example mpl plot added
      jsplot example added
      runserver.py added
      Hook it up with jsplot
      Add a screenshot
      Better README added
      README improved
      Show some testing data
      Use better testing data
      License added
      setup.py added
      Use testing data if data == []
      Just return from the function on CTRL-C
      spkg-install added
      prepend the .. path, instead of append
      Use the proper local files in ./manage.py
      Turn off points
      Fix a bug
      Add grid and legend

I fixed some scipy warnings in hermes2d:


ondrej@crow:~/repos/hermes2d(master)$ git weekreport 
Ondrej Certik (4):
      hermes_common updated
      Fix the other SciPy warning in Python wrappers
      Update the generated .cpp file
      update hermes_common

I have fixed Python wrappers in hermes1d, made the Schroedinger example work again:


ondrej@crow:~/repos/hermes1d(master)$ git weekreport 
Ondrej Certik (19):
      Use jsplot if available
      Build everything for now
      Make the Python wrappers work again
      Fix the lhs/rhs/residual to build properly
      Polish the forms a little bit
      Enable assembling
      Implement c2py_Mesh
      Use c2py_Mesh in the schroedinger example
      Update insert_matrix in Schroedinger
      Fixed the rest in Schroedinger
      copy_mesh_to_vector() and copy_vector_to_mesh() added to the Mesh class
      Use JSPlot
      Linearizer.get_xy() fixed
      Implement Mesh.copy_vector_to_mesh() and use it from get_xy()
      Use CooMatrix to assemble
      Implement pysparse solver
      Polish the numpy solver
      Polish the printing a little bit
      Rename assemble_jacobian() to assemble_matrix()

I fixed some build issues with FEMhub:


ondrej@crow:~/repos/femhub(master)$ git weekreport 
Ondrej Certik (3):
      Add gnutls dependency for Python
      opencdk added
      Fix libgpg_error to build in parallel

and I have some more patches to the build system in my "pu" branch at github:


* 44398d8 (github/pu) Use absolute paths in "femhub -i"
* 2b17bf0 Fix "femhub --devel-install" to normalize the paths
* 3b41261 Make "femhub --shell" stay in the current directory

I still need to test that femhub builds fine with those (it always takes quite some testing to make sure it builds properly).

week June 5 -- June 11

2010-06-11T23:44:00.000-07:00

On Sunday I moved to Livermore.

On Monday I rebased my patches for Euler Equations, DG and FVM for hermes2d, sent for review. Unfortunately it took till Friday until they got reviewed, but they are now in.
I couldn't sleep, so at around midnight I turned on my computer and rewrote the build system for femhub, it took me about 1.5h. Then I could finally sleep well.

On Tuesday I had my new hire orientation. During it I wrote couple more patches for femhub.

On Wednesday I got my badge and spent the rest of the day with paperwork. They didn't manage to get my computer setup, so I studied a bit of Quantum Field Theory from a book that I found in our room and then spent the rest of the night figuring out the weak formulation for the radial Dirac equation. So far no luck, but at least I wrote it in many different forms in my hand written notes. Sometimes it's not bad to be cut off the internet and computers.

On Thursday I got my computer account setup and spent most of the day with paperwork.

On Friday I finally did some real work. Then we got to a pub, got home and drank the rest of my Budvars (Czechvar) and Plzeňs (Pilsner Urquell).

The FEMhub build system is really cool. It's written in Python and I totally got rid of the old Sage build system. Some features of the new one:

* dependencies
* automatically uses all your processors (unless told otherwise)
* simple Python script (414 lines) that handles everything, plus sage-env, sage-spkg and sage-make_relative scripts (the rest I simply deleted)
* allows you to install just some packages, for example "./femhub -i python" just installs python and it's dependencies
* allows you to unpack any package into the devel/ directory and then build it

I will not use it to develop hermes1d+schroedinger and dirac solvers over the summer, so it will likely get improved in the coming weeks. (I have to use it, because my computer is some old RHEL5, and I don't have a root access. With FEMhub, I have all the Python libraries plus cmake and similar stuff right there. It's really awesome.)

In total, I wrote 52 patches for FEMhub, 3 for hermes_common, 9 for hermes2d, 8 for hermes1d and 1 for sympy:


FEMHUB:
Ondrej Certik (52):
      scipy and python upgraded
      Remove the old buildsystem
      New Python based buildsystem implemented
      Install hermes2d by default
      Compile in parallel
      Let CmdException propagate
      Handle dependencies
      Fix the PYTHONPATH issue
      Add a check that IPython is installed
      Add Cython dependency to hermes2d
      Add matplotlib to hermes2d deps
      Keep track of installed packages
      Add the -j option
      Add the rest of the packages
      Disable sphinx for now
      Add a todo item
      Disable mayavi for now
      Add pysparse to fipy deps
      Change the banner
      Fix a typo
      Remove old .hg files
      setuptools added into fipy's deps
      lab() implemented
      Don't let other import errors to pass silently
      Build bzip2 before Python
      Simplify the makefile
      Add the jinja2 package
      Remove one forgotten " from a message
      --shell and -s/--script options implemented
      Fix cmake so that it builds if old cmake is present
      expandvars() added to cmd()
      Remove the rest of old buildsystem files
      Move spd imports to femhub imports
      Move download_packages into the femhub buildsystem
      Remove a forgotten file from the old buildsystem
      Create the standard directory if it doesn't exists
      Polish the banner appearance
      femhub-run: little refactoring
      Use proper ipythonrc and matplotlibrc
      Wrap lab(), add debugging statements
      Don't import lab() in the ipythonrc
      Fix the missing "-p" when creating the standard directory
      Fix the problem with ambiguous names
      cpu_count refactored
      hermes1d added
      -f/--force option added
      --unpack added
      --pack option implemented
      --devel-install option implemented
      Use MAKEFLAGS instead of MAKE env variable
      Improve the FEMhub shell prompt
      Print info when unpacking


hermes2d:
Ondrej Certik (9):
      Removing the old-code directory
      hermes_common update
      Reformat the documentation in the Geom class
      Python wrappers updated
      Regenerate the _hermes2d.cpp
      plot.py: ScalarView updated
      Element orientation exported
      Sanity checks and docs
      Add all shapefunctions to the space L2

hermes1d:
Ondrej Certik (8):
      spkg-install added
      "make install" implemented
      Make it build on the Mac
      Allow to turn off examples
      Use std::max(), fixed several warnings
      hermes_common updated
      Fix h1_polys.py to work with FEMhub's SymPy
      Fix the new line warning

hermes_common:
Ondrej Certik (3):
      Add numpy2c_double_inplace() to _hermes_common.pxd
      Update the generated cpp file
      Fix a small double -> int bug/warning

sympy:
Ondrej Certik (1):
      pyglet: fix string exceptions

week May 30 -- June 4

2010-06-04T14:59:00.000-07:00

Over the weekend and Monday I went to Livermore, CA to rent a room to live. On Monday and Tuesday my uncle visited me from Prague so I showed him Virginia City and Lake Tahoe, that was lots of fun.

Then I tried to debug hermes1d and some matrix issues in there for couple hours, but failed so far.

I've helped Sameer to fix h5py to compile with FEMhub.

Yesterday I spent 12h setting up gitosis on our server and finally got it done, here is the web interface:

http://git.hpfem.org/

I had to patch the gitosis repository itself, here are my changes: http://github.com/certik/gitosis (I've also sent them to Tommi Virtanen, the gitosis author, but I haven't heard back yet.)
In case you wanted to install it as well, I've posted instructions here.

I've figured out how to use msysgit with windows and use ssh keys to login to a linux box. It turns out it's not so trivial, see this issue.

I submitted the following patches to our projects at hpfem.org:


ondrej@raven:~/repos/hermes1d(master)$ git weekreport 
Ondrej Certik (3):
      Reverting the change dfa8580
      hermes_common updated
      hermes_common updated

ondrej@raven:~/repos/hermes2d(master)$ git weekreport 
Ondrej Certik (5):
      Fix "make install" to work again
      hermes_common updated
      remove hermes_common/doc/Makefile from .gitignore
      Remove _XOPEN_SOURCE and _POSIX_C_SOURCE hacks
      Fix the compilation warning

Btw, my "git weekreport" alias is defined as:


[alias]
    weekreport = shortlog --since=1.weeks --author=ondrej

Next week I am moving to Livermore, and my summer internship will get started.

SymPy GSoC has started

2010-05-26T17:15:00.000-07:00

This summer SymPy got 5 students, you can see their proposals and blogs here:

http://code.google.com/p/sympy/wiki/GSoC2010

All are required to blog once a week, the deadline is Friday night PST, and I'll do so too.

This summer I will be at the Lawrence Livermore National Laboratory, applying hp-FEM to radial Schroedinger and Dirac equations as well as 1D Density Functional Theory. I need to check with the management there how much I am allowed to blog about it. If I am, I'll keep blogging once a week too about it. If not, then only about sympy. I am super excited about the job, as it is in the electronic structure field, which is what I always wanted to do.

ESCO 2010 conference

2009-12-13T19:38:00.000-08:00

An interesting conference 2nd European Seminar on Coupled Problems (ESCO) will be held on June 28 — July 2, 2010 in Pilsen, Czech Republic.
Among the topics are solving PDEs and applications and using Python for scientific computing. In particular, Gaël Varoquaux is the keynote speaker.

Unfortunately, it was later announced that the SciPy 2010 conference is going to be at the same time, which is really unfortunate. But here are some reasons why you should consider going to ESCO 2010 instead:

If you like numerical calculations (finite elements, differences, volumes, ...) and solving partial differential equations and other problems and also programming in Python, together with C/C++ or Fortran, you will have a chance to meet some of the top people in the field. SciPy conference usually has people who solve PDEs (e.g. SciPy 09 had about 6), but ESCO 2010 will have about 60. So ESCO wins.

Robert Cimrman, who you probably know from the scipy and numpy mailinglists, also the author of the sfepy FEM package in Python, lives in Pilsen, so he'll gladly show you some good Pilsen pubs. SciPy 2010 is going to be in Austin and while Austin has cool pubs too, I must be fair and I liked that (I was there at the Sage 08 days), but it's just not comparable, the beer is better in Pilsen, it's a historic city and there are more pubs.

Pilsen is close to Prague, so you will have the chance to visit it. You should walk in the old town, have couple beers etc. Here you can see some photos that Gaël took when we met in Prauge. Again, this is incomparable with Austin.

It is held in the Pilsner Urquell Brewery. When Pavel Šolín announced that at the SciPy 09 conference, Jarrod asked "Ah, in a beer pub?". So let me be clear. The word pilsner (type of the beer) is coming from the Czech city Pilsen (Plzeň in Czech). Pilsner Urquell is not some beer pub (e.g. even Reno where I live now has a beer pub), it's The Brewery. Austin is a cool place (and Texas steaks are really good), but as you can see now, it absolutely cannot compete with Pilsen.

If you have time, I can fully recommend to go to ESCO 2010.

SciPy 2009 Conference

2009-08-24T22:18:00.001-07:00

I attended the SciPy 2009 conference last week at Caltech.

When I first presented about SymPy at the scipy 07 conference exactly 2 years ago, it was just something that we started, so noone really used that. Last week, there were already 6 other people at the conference who contributed one or more patches to SymPy: Robert Kern, Andrew Straw, Pauli Virtanen, Brian Granger, Bill Flynn, Luke Peterson.

I gave a SymPy tutorial, main presentation and Luke gave a PyDy + SymPy lightning talk (seek to 16:04).

I also gave my experience with designing a traits GUI for FEM lightning talk (seek to 5:35).

My advisor Pavel Solin gave a talk about Hermes and FEMhub and other things that we do in our group in Reno.

Besides that, it was awesome to meet all the guys from the scientific Python community, meet old friends and also get to know other people that I only knew from the lists. I was pleased to meet there people who solve PDE using Python, we had many fruitful discussions together, and as a result, I already created FEMhub spkg packages for FiPy, other will follow. Our aim is to create a nice interface to all of them, so that we can easily test a problem in any PDE package and see how it performs.

Overall, my impression is very positive. I am very glad I chose Python as my main language many years ago, all the essential parts are now getting there, e.g. numerics (numpy, scipy, ...), symbolic (sympy, Sage, ...), 2d plotting (matplotlib, ...), 3d plotting (mayavi), GUI (traits UI, which supports both GTK, QT on linux and native widgets on Mac and Windows, and Sage notebook for web), excellent documentation tool with equations support (Sphinx), lots of supporting libraries, like sparse solvers and then very easy way to wrap C/C++ code using Cython and to speed up critical parts of the Python code using Cython. It's not that each of those libraries is the best in the world --- in fact, not a single one is --- but together as an ecosystem, plus the high level of (free) support on the lists for all of those libraries, this in my opinion makes Python a number one choice for scientific computing, together with C, C++ and sometimes Fortran for CPU intensive tasks and/or legacy libraries.

SymPy on the google phone

2009-08-22T19:08:00.001-07:00

Here is a proof that sympy works on the google phone:

Thanks to Rae S. Yip from Caltech and Nicolas Pinto for taking the picture.

Los Alamos Sprint

2009-08-15T16:32:00.001-07:00

Last weekend, Luke and Aaron came to visit me here in Los Alamos and we accomplished the following:

* documentation doctests fixed
* pexpect wrappers to maxima
* couple match bugs fixed
* lots of patches reviewed and pushed in
* made pydy simplify trig expressions
* pexpect wrappers to autolev
* work on the odes module
* visited hot springs

Pictures of the last item were requested, so Aaron (left), Luke:

Aaron, Ondrej:

Ondrej, Aaron:

My experience with running an opensource project

2009-05-10T12:14:00.000-07:00

Nir Aides, the author of the excellent winpdb debugger, sent me the following email on September 21, 2008, so I asked him if I can copy his email and reply in form of a blog post (so that other people can comment and join the discussion) and he agreed. It took me almost a year to reply, but I made it. :)

Hi Ondrej,

How are you?

I am about to publish a new free software project, a new simple PHP framework, and I am interested in your advice.

You started SymPy and were able to make other people join you and develop it with you.
How did you do it?
How did it happen?
Did you actively call for other people or they spontaneously showed interest and joined you?
Are the other major contributor people who were your friends before you started the project?
Did you need to create or manage the project in a particular way to make it attractive to other people?
Are there things you are aware of that promote collaboration or demote it?

I was never successful in doing the same with Winpdb, which while it became reasonably popular, no one has ever joined me to develop it, except for a notable tutorial contribution by Chris Lasher which was developed independently.

Now with the new project, I am wondering what are my chances of making other people try it and take it on. On the one hand it is a new and fresh code base in an interesting field, on the other hand, why would anyone bother to spend their energy on this new project when they have Symfony or Drupal?

What do you think?

BTW, Ohloh believes you have a median of 19,000 lines of changed code per month since the start of their log. Can this be true? Is this humanly possible? According to it SymPy has over 1,000,000 lines of code? I can't understand these numbers. Winpdb has about 25,000 lines after 3 years of development. And from my experience 1,000,000 lines of code projects need about 20-50 full time developers to work on for 2-5 years which is about 40-250 man years. And as if this is not enough you are listed as owner in a dozen other projects in Google code and have enough time to become an awarded scientist. How is this possible?

http://www.ohloh.net/p/sympy/contributors/

BTW2, do you still use Winpdb? If you find yourself using it less, can you say what are the reasons, or what it would take to make it more useful?

BTW3, How is SymPy doing?

Cheers,
Nir

So my most honest answer how to run a successful opensource project is: I don't know.

But nevertheless I tried to summarize some of my ideas and experience and some guidelines that I try to follow, maybe it will be useful to you Nir, or anyone else.

First of all, there has to be a public mailinglist (easily accessible), public bug tracker, nice webpage, easy to find downloads, frequent releases (once a month is good, but in the worst case at least 4 times a year) and a set of guidelines to follow in order to contribute. So that's a must, if the project doesn't have the above, it's almost impossible to become successful. However, that is just a start, just a playground. There are still many projects that have the above and yet they totally fail to attract developers.

So I think the most important principle is that I always think how to employ other people in what I do. If I have some plan in my head how to do something, e.g. how to move some things forward, I always create exact steps and put it to issues, or our mailinglist, so that each step can be done by someone who is completely new to sympy. So I try to look at things from other people's perspective and think -- ok, I quite like this SymPy project and I'd like to get this done (for example a new release, or something fixed, or implemented), but I have no idea how to start and what exactly needs to be done.

So what I try to do if someone comes to our list and asks for something, is that I create a new issue for it and think how I would fix it if I had time. Then write the necessary steps in the issue and invite the submitter to fix it and I offer help with explaining anything and guiding. Now there are two things that can happen. Either the submitter has time and a will to go forward and in this case he starts wrestling with it and whenever he has some code or a question, I need to find time, review it and offer some way out. Or the submitter is too busy, in which case the instructions simply rest in the issues and the next time someone asks for the feature, the instructions are already there. I don't have estimates how frequent either case is.

When I am working on something myself, I try not to code privately, but also put up issues first and put the steps needed in the issues, so that it's easy for other people to join in.

In general, the most precious value for me is the fact that someone else had to sit down at his computer and wrote the patch. So I do everything possible to get new (or more) people interested in the development. Some people think that only super programmers can do a decent job and it's useless to invest time in people that may just have started with Python. They are wrong. Among the SymPy developers (around 65 people total have contributed patches so far at the time of writing this post), we have all kinds of people. We have people from high school, we have a retired US army engineer, we have physicists, mathematicians, biologists, engineers, teachers, or just hobbyists, who do it for fun. Unfortunately, we do not have many women (I think no patch that made it into sympy was contributed by a woman, but I may be wrong), so if anyone has any ideas how to get more women involved, let me know (I know we have several women fans, so that's a good start:). We have people whose first open source project they ever contributed to was sympy and people who are new to Python.

Many times the first patch that a new potential developer submits is not perfect, usually it's faster for me to write it myself, than to help with the first patch, however my rule is to always help the submitter do that. Sometimes he sends a second patch, or a third, and usually it needs less and less work on my side and it already pays off, because he is then able to fix things himself, if he discovers a bug and sympy has just won a one more contributor.

So I came to the conclusion that all that is needed is an enthusiasm. You don't even have to know Python (as you can learn all these things on the way) and you can still do useful things for us and really spare our time.

To answer another question from Nir's email, SymPy has about 130000 lines of code and another about 20000 lines of tests, so I think those stats are wrong. Also the changed lines of code is in my opinion wrong, we usually have about 250 new patches per release (this depends how often we release and other things).

Yes, I am involved in couple other projects, e.g. Debian, Sage, ipython, scipy, hpfem.org (and couple more), basically everything that has to do with numeric simulation and Python, but my activity there varies. The most time consuming thing in the last couple years was definitely school, I was finishing my master in Theoretical Physics in Prague and then moved to the Nevada/Reno and I just finished my first semester here at PhD in Chemical Physics, and sometimes it was just crazy, e.g. I finished teaching at 7pm and instead of going home and sleep, I stayed in my office, fixed 10 sympy issues that were holding off a release, finished at 1am, went home (by bike, since I don't have a car yet), slept couple hours and then did just school again for a week, other people reviewed the issues in the meantime, and then I made the release (instead of sleeping again). In the last semester it was not unusual that I got home at 1am every week day, then slept most of Saturday to catch up, on Sunday I did some laundry and shopping, and the rest of time I did grading and homeworks for all my classes and teaching, no time for anything else (e.g. no friends, no girls, no rest, no hobby, no opensource stuff, nothing). So sometimes one has to work pretty hard to get through it, but fortunately it's behind me finally, if all goes well, I should be just doing research from now on and have a real life too. Also I am sorry I didn't manage to reply sooner. :)

To answer the other questions:

Are the other major contributor people who were your friends before you started the project?

No, not a single major contributor was my friend before I started the project. Every single one of them become a developer using the procedure I described above, e.g. first showed on the list or in the issues, and maybe even the very first patch was not a high quality one (and if I was stupid and arrogant, or didn't see the big potential, I would just ignore them). But when given a chance, they became extremely good developers and sympy would simply just not be here without them.

Did you actively call for other people or they spontaneously showed interest and joined you?

I very much encourage everyone to contribute, but the initial interest must be in them, e.g. they at least have to show around the mailinglist/issues, so that I know about them. But once I know they are interested in some issue, yes, I try to invite them to fix it, with my help.

One observation I made is that I have to always think in the spirit "how to earn new money, not how to spare the money I already have", e.g. when applied to sympy, how to get new developers, how to develop the new great things etc. Even if I am super busy as I was, I still have to think this way. Once I start thinking how to conserve and preserve what we already have, I am done, finished and that's the road to hell.

If I am open, positive, full of energy, I can see people joining me and we can do great things together. It probably sounds obvious, but it was not for me, when for example some people I worked with, started their own projects, when I got busy, and started to compete, instead of helping sympy out. And I felt betrayed, after so much work that I invested into it and started to become protective. And then I realised that's wrong. I can never stop other people do what they want to do. If they want to have their own project, they will have it. If they don't want to help sympy out, they won't (and what is more important, there is nothing wrong with either of that). It's that simple and being protective only makes things worse.

There is also a question of the license that you use for the project, e.g. one should basically only choose between BSD (maybe also MIT or Apache), LGPL and GPL (there are also several versions of the GPL licenses). Unfortunately the fact is, that there are people who will never contribute a code under a permissive BSD license (because it's not protecting their work enough) and there are also other people who really want to code to be BSD (or other permissive license) so they can sell it and they don't need to consult with lawyers what they are or aren't allowed to do and also so that they can combine it with any other code (opensource or not). It also depends if one wants to combine (and distribute) other codes together. So choosing a license is also important. I believe that for sympy BSD is the best and for other projects (like Sage) GPL is the best and one has to decide on a case by case basis. For Winpdb, I would make it BSD too, since you can get more people using it.

To conclude, SymPy is a little more than 2 years old, and it has been a great ride so far and more things are coming, e.g. this summer we have 5 Google Summer of Code students and people are starting it to use in their research and we plan to use it in our codes at our group here in Reno too, so things look promising. I am really glad, we managed to build such a community, so that when I am busy, as I was the last semester, other people help out with patches, reviews and other things, so that the project doesn't stall and when I got rid of my school duties now, we can move things forward a lot.

So maybe you can get inspired by some of the ideas above. I am also interested in any discussion about this (feel free to post a comment below, or send me an email, or just write to a sympy list about what you think).

Newtonian Mechanics with SymPy

2009-03-17T14:12:00.000-07:00

Luke Peterson from UC Davis came to visit me in Reno and we spent the last weekend hacking on the Python Dynamics package that uses SymPy to calculate equations of motion for basically any rigid body system.

On Friday we did some preliminary work, mostly on the paper, Luke showed me his rolling torus demo that he did with the proprietary autolev package. We set ourselves a goal to get this implemented in SymPy by the time Luke leaves and then we went to the Atlantis casino together with my boss Pavel and other guys from the Desert Research Institute and I had my favourite meal here, a big burger, fries and a beer.

On Saturday we started to code and had couple lines of the autolev torus script working. Then we went on the bike ride from Reno to California. I took some pictures with Luke's iphone:

Those mountains are in California and we went roughly to the snow line level and back:

This is Nevada side:

That was fun. Then we worked hard and by the evening we had a dot product and a cross product working, so we went to an Irish pub to have couple beers and I had my burger as usual.

On Sunday we spent the whole day and evening coding and we got the equations of motion working. On Monday we worked very hard again:

and fixed some remaining nasty bugs. I taught Luke to use git, so our code is at: http://github.com/hazelnusse/pydy, for the time being we call it pydy and after we polish everything, we'll probably put it into sympy/physics/pydy.py. If you run rollingtorus.py, you get this plot of the trajectory of the torus in a plane:

It's basically if you throw a coin on the table, e.g. this model takes into account moments of inertia, yaw (heading), lean, spin and the x-y motion in the plane. Depending on the initial conditions, you can get many different trajectories, e.g for example:

or:

This is very exciting, as the code is very short, and most of the things that Luke needs are needed for all the other applications of sympy, e.g. a good printing of equations and vectors (both in the terminal and in latex), C code generation, fast handling of expressions, nice ipython terminal for experimentation, plotting, etc.

Together with the atomic physics package that we started to develop with Brian sympy will soon be able to cover some basic areas of physics. Other areas are general relativity (there is some preliminary code in examples/advanced/relativity.py) and quantum field theory and Feynman diagrams - for that we need someone enthusiastic that needs this for his/her research --- if you are interested, drop me an email, you can come to Reno (or work remotely) and we can get it done.

My vision is that sympy should be able to handle all areas of physics, e.g. it needs good assumptions (if you want to help out, please help us test Fabian's patches here), then faster core, we have a pretty good optional Cython core here, so we'll be merging it after the new assumptions are in place. Then sympy should have basic modules for most areas in physics so that one can get started really quickly. From our experience so far in sympy/physics, those modules will not be big, as most of the functionality is not module specific.

SIAM 2009 conference in Miami, part I

2009-03-05T20:35:00.000-08:00

I am at the SIAM Conference on Computational Science and Engineering (CSE09) and it is awesome. Right now, I am writing from the 50th floor with Pearu Peterson (f2py), Brian Granger (ipython), Fernando Perez (ipython) and John Hunter (matplotlib), I took videos of them and with their permission, posted to youtube. The view from the balcony is spectacular. My own room used to be in the 15 floor in the Hilton hotel and I thought man, this is high, but then I visited Fernando and John in the 50th floor in their apartment and our 20 story hotel looks like a small hut.

As usual, I met lots of old friends and made some new ones. I liked the electronic structure section on Wednesday and a Python section today. I was also working very hard to get mayavi2 working in Sage to be ready for my presentation and it seems I just finally made it.

Experience with git after 4 months

2008-12-23T17:03:00.000-08:00

I switched to git from mercurial about 4 months ago. As Matt Mackall (the main author of mercurial) has pointed out, my arguments were partially just excuses for the switch, because everything could be fixed in mercurial as well (eventually). On the other hand, I really wasn't expecting to switch when I invested time and energy to learn mercurial. Also because I thought that git doesn't have such a nice documentation, has a steep learning curve and the community is hostile, because it is developed by the kernel people who are famous for sometimes being not so nice on the mailinglists (as I now found out, none of these are true). I was getting annoyed by small things one by one, after finally I said to myself enough is enough, let's just switch.

So as to the steep learning curve --- it took me a day or two to being able to do what I want, so it seems to me it's really easy to learn once you know some distributed vcs. I found out the documentation is much better than in mercurial --- in git one just types "git help command" and a full man page fires up with lot's of information and examples. In mercurial, one just gets a couple lines (compared to git) of help in the terminal, without examples. Also the first (actually second) thing I noticed is that "git log" (and other commands like "git diff") doesn't scroll out my terminal, but uses "less" automatically -- that's just great. The first thing I noticed is that git is superfast for almost every operation, especially it's noticeable with "git diff". As to the community, I only had a chance to send a few emails to the git list, so I cannot really say, but so far it's very responsive and friendly.

However, those are just little things that can (and I am sure they will) be fixed in mercurial as well. What is a big thing are branches, especially remote branches. One just fetches a remote repository and then works with it like with any other local branch, e.g. one can switch to it, or just "git log some_remote_branch" to see what is in there. One can easily compare them etc. With mercurial, I was using "hg out" and "hg in" commands to see what changes I will pull or push, but those commands require internet connection, so it really sucks and it's slow. In mercurial, I was using different directories for different branches, but that's just extremely inconvenient and also slow (creating a new branch is just a matter of adding one file, not copying the whole repository --- for example with the sympy hg repository, I often had to wait several seconds to clone a repo, while with the git repository it's just instant) and big (in terms of megabytes).

The bible of Mercurial, the hgbook says:

In most instances, isolating branches in repositories is the right approach.

But I think it's the wrong approach. The only argument for doing this that I accept is that sometimes you are not sure which branch you are in in git --- well, I use this trick (actually, just read the documentation of __git_ps1 in /etc/bash_completion.d/git), so my bash line always says which branch I am in using the red color. I never do any mistakes with this.

Anyways, I could continue like this (e.g. check out "git svn", "git rebase -i" and many other things), but you can read why git is better for example here, no need to repeat here. I am very happy now and I can only recommend to learn git.

So let me also say some things that are worse in git --- one thing that could be improved are URLs of the gitweb (hgweb urls are neat, gitweb urls use "?id=..." stuff). Another thing is that the debian package git is not git! You need to install git-core to get git (but this seems to be getting fixed).

There is also an interesting discussion happening right now in the debian-python mailinglist about switching the DPMT repositories to git. I think almost all opinions (for and against) were already stated, but if you have some opinion that wasn't yet said, please do so.

Google Mentor Summit III

2008-10-28T08:23:00.000-07:00

On Sunday I checked out from the hotel and enjoyed the breakfast, basically the same as the first picture here. Which is cool, the best way to start a day is with a good breakfast in the morning.

At Google I finally met with Bart Massey, who works at the Portland State University where I was an intern (in 2005) and also he was sponsoring one SymPy GSoC student in 2007.

Then we started to port SymPy to Jython with Philip Jenvey. I first installed openjdk-6, which is in Debian main, then checked out Jython svn, typed "ant" and it just worked. This is so awesome, that we finally have a truly opensource and fully working java implementation in Debian. Philip fixed some bugs already, so SymPy imports just fine. Most of the tests run, but also quite a lot still fail.

We managed to isolate two bugs: issue 1158 and 1159. Then there was an annoying recursion bug, that took us several hours to dig into and I then had to leave at 6pm without a clue. But Philip kept saying, I just need to look more at the java stacktraces that Jython was generating and I'll figure it out. And he did! The next morning he filed an issue 412 in the PyPy tracker, because they seem to have the same problem. There are subtle differences in how to handle __mul__ and __rmul__ methods that are not well documented, so it works in CPython but fails in Jython/PyPy. I think we can fix that in SymPy too in the meantime, so I am very excited, because it seems SymPy will work on top of Jython soon.

There was also a git vs hg session with both hg and git developers. Since I recently switched from hg to git for all stuff where I can choose the vcs, it was very interesting for me. And I must say I am glad I switched. Mercurial works fine enough and with the recent rebase feature I think it is usable, but git just has a bigger momentum at the moment and it just doesn't get in my way.

During the day Steve sent an email to the GSoC list that Dirk forgot his jacket in the hotel room, so I said to myself, haha, that's a pity. And then I realized, oh shit, I left my jacket there as well. They didn't have it in the evening when I stopped there, but fortunately they sent me an email yesterday that they found it, so I'll stop there on my way back.

We also took couple of group pictures, here is just my lame attempt (I am sure people will soon post better ones):

On Monday morning I met with Fernando Perez, the author of ipython, he know works with Jarrod. Then I rented a car and came to visit Brian Granger and his family in San Luis Obispo.

We went to the beach in the evening:

And had a dinner together. Brian showed me his options pricing example he did with SymPy, so I made him create an issue 1175 with the code.
Then we played with the parallel stuff that Brian contributed to ipython and we tried sympy with it and to our disappointment, we found pickling bugs in sympy, it works with protocol 0, but not protocol 2, see the issue 1176 and 1177. This almost smells like a bug in Python itself, but I need to investigate that later more deeply.

Today is Tuesday and we'll work on doing atomic physics calculations with sympy. See also Brian's email to the sympy list about it.

Google Mentor Summit II

2008-10-26T08:47:00.000-07:00

On Saturday morning I met Steve McIntyre and Dirk Eddelbuettel, Steve took us by car to Google. Well, it was overwhelming.

For example we met there two other Debian developers, Jeff Bailey and Daniel Burrows. There are developers from all the famous projects, like git, Wine, Turbogears, Jython, ... There are still people I wanted to meet but didn't manage yesterday, need to fix that today. My plan is to get SymPy running on top of Jython, or at least do some more progress to take the advantage, that Philip Jenvey from Jython is here. Also I wanted to fix some RC bugs in Debian when Steve is here, to make some progress on my NM finally. We'll see how it goes.

There was also an excellent presentation from people behind Android. One thing I was curious if it's going to run native C applications, like Python and it seems it will happen eventually, all the sources are out there, so someone just needs to push it forward. Another very cool thing they did that I was always looking for is Gerrit for reviewing patches (see review.source.android.com for example how it looks like) -- basically a fork of Guido's codereview, but it has many nice features, like automatically applying the patch to the git repository as a new topic branch and one can then easily pull it, as well as review it over the web interface. When I get back to Prague I am going to install this for SymPy.

Google Mentor Summit I

2008-10-25T07:24:00.000-07:00

I flew via Atlanta and had only about an hour to my next flight to San Francisco, so after my last experience, when I went to the immigration, got stuck in the line for more than an hour and then had to run to catch my flight, I decided to try a different strategy this time: first run to the immigration and then walk to my gate. Unfortunately I was sitting near the back of the plane and I got out among the last ones. Fortunately, it was several hundreds meters to the immigration, so I run as fast as I could and I managed to get there as the first one and everything took about 5 minutes. That was just awesome, I finally figured this out.

Jarrod was waiting for me at the airport, went to his place. Here's his cat:

On Friday we did some work and then went to the Golden Gate, the traffic was quite dense:

Alcatraz:

San Francisco:

Then we had a cofee in San Francisco and went to Silicon Valey, Jarrod drove me around a little bit to see SLAC, Stanford campus and other things. In the evening we went to the common pub with other mentors and Google guys, where we for example met with Robert Bradshaw.

Today I am looking forward to meet with all the people I know from mailinglists, Debian and other places.

Gael and Emmanuelle in Prague

2008-10-01T12:19:00.000-07:00

Last weekend Gaël Varoquaux together with Emmanuelle came to Prague, so it was really awesome to meet again (we first met with Gaël at the SciPy 2007 conference at Caltech).

Because I recently finished my master, I was basically visiting pubs each evening, so we first met on Friday at 10pm and just had couple Pilsens (I already had some beers that evening with other friends), then I had to go, as I was doing TOEFL on Saturday. At the exam I met a very beautiful girl and other interesting people, so we went to lunch together and as a result I arrived half an our later than we agreed with Gaël and Emmanuelle, but I think they understood my situation. :) Then we went around Vyšehrad, came to I.P.Pavlova, had some Czech meal and beers, then went on foot pass the Church of Saints Cyril and Methodius, where the Heydrich attackers were cornered (there is still the bullet-scarred window visible from the street), then continued over the bridge to the hotel, Emmanuelle went in, I then came with Gaël to another pub for couple of more beers.

The Python scientific community is very cool and I always enjoy meeting people from it and discussing things like cython, scipy, ipython, mayavi, sympy, matplotlib, sage, what license is best for each project, etc. in Prague pubs. Python has a lot of high quality libraries and tools for scientific computing, so things look very promising.

It was fun I really enjoyed that.

master studies

2008-09-22T08:59:00.000-07:00

I did it! :) I defended my thesis and passed master finals from theoretical physics at Charles University in Prague couple hours ago.

After almost dropping out of my school exactly a year ago for not having enough credits to go to the next year, I gave myself an obligation to finish my school on time. I worked very hard the last year, I had to do 8 exams, some of them very hard, requiring more than a week of thorough learning and 9 seminars, requiring a lot of work too and also a master thesis, for which I had to had working finite element solvers and together with SymPy it took all my time and energy. I even had to cancel my trip to Austin and Caltech for the SciPy conference. But I finished my school after all, I am very happy about it.

Last year, two of my friends bet $100 between themselves that I will not finish on time. Jarda, who believed in me is now at Princeton doing his Ph.D. Matouš, who didn't believe in me, will now pay $100 to Jarda. I think that life is fair.:)

Now I'll be visiting pubs quite often and then I'll fix some long standing issues in SymPy, hopefully finish my Debian task & skills (to finally become a Developer couple months after that) and finally do useful stuff for my research with a fresh head now.

I am switching from Mercurial to git

2008-08-16T23:14:00.000-07:00

After a long night of debugging and while preparing, reviewing and pushing final patches before a SymPy release, I got this:


$ hg qpush 
applying 1645.diff
Unable to read 1645.diff
** unknown exception encountered, details follow
** report bug details to http://www.selenic.com/mercurial/bts
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 1.0.1)
Traceback (most recent call last):
  File "/usr/bin/hg", line 20, in 
    mercurial.dispatch.run()
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 20, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 29, in dispatch
    return _runcatch(u, args)
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 45, in _runcatch
    return _dispatch(ui, args)
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 364, in _dispatch
    ret = _runcommand(ui, options, cmd, d)
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 417, in _runcommand
    return checkargs()
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 373, in checkargs
    return cmdfunc()
  File "/var/lib/python-support/python2.5/mercurial/dispatch.py", line 356, in 
    d = lambda: func(ui, repo, *args, **cmdoptions)
  File "/var/lib/python-support/python2.5/hgext/mq.py", line 1942, in push
    mergeq=mergeq)
  File "/var/lib/python-support/python2.5/hgext/mq.py", line 833, in push
    top = self.applied[-1].name
IndexError: list index out of range

I am using:


$ hg --version 
Mercurial Distributed SCM (version 1.0.1)

Copyright (C) 2005-2008 Matt Mackall  and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And I just got pissed off and I am switching to git for good. I've been using mercurial every day since about November 2007, and I think I am quite proficient in it. But we were missing some features, like recording a patch by hunks (git add -p), so Kirill Smelkov (another SymPy developer) just implemented this to mercurial queues, it is now included in hg 1.0. But I am constantly beaten by stupid bugs, mercurial broke into pieces several times already (last time with hg bisect, before it with mercurial queues and file renaming), that just should not happen in a production version. Another problem is that hgweb is constantly using 100% of CPU on my server whenever someone clicks to see the contents of the README file. Again I can spend time debugging it, but I will just use git and if it works, I'll stay with it.

I started learning git recently and it is just better in every aspect. Bigger community, it has all the features that I was always missing in mercurial (like rebase, hunks recording, diff diff --color-words, better gitk). And it's superfast.

Unless I find some showstopper bugs in git too, I don't think I am coming back.

We'll have to create a live mercurial mirror of SymPy, so that people can continue using mercurial, but I'll myself just be using git. We wrote a simple translation table for anyone using mercurial to get started:

http://wiki.sympy.org/wiki/Git_hg_rosetta_stone

I love Python and I like that Mercurial is just a small C core and the rest is in Python, but for some reason, it still has the baby bugs, that should have been fixed years ago and why should we spend our time fixing and improving Mercurial, if other people have already done the job for us (and usually a better job than I could do) in git?

Another reason for switching is that in Debian, mercurial is not used for packages, while git is used a lot.

SymPy core in Cython, general CAS design thoughts

2008-08-14T14:47:00.000-07:00

We managed to write a new core of SymPy that is an order of magnitude faster than the current sympy, but contrary to sympycore (which is Pearu's attempt to write a fast core), it uses the same architecture as sympy, so we can now merge it back and speedup everything.

It started like this: I was unsatisfied with the current sympy speed, and also with the fact, that it was only Kirill and sometimes Mateusz who were submitting speedup patches to sympy, but unfortunately noone was doing any effort to get sympycore merged. I (and I think many people too) hoped that when sympycore got started, the aim was to get it merged back. That was not happening. So I said to myself, ok, noone does the job for you, you need to get your hands dirty and gets things moving yourself.

I came with my thoughts 3 years back, when it was a rainy weekend in Portland, Oregon, I was sitting in my office at the university, just finished playing with swiginac that we wrote with Ola Skavhaug, and said to myself then -- no, there must be a better way. First all this bloated swig thing, but this could be fixed by hand (today we just use Cython). But especially with how things were done in GiNaC internally. Too many classes, too difficult to write new functions. So I wrote my own cas in pure Python just to see if things can be done better. Then a year later at the Space Telescope Science Institute, I was again wondering, ok, a good CAS in Python is really needed.

So I wrote to ginac developers saying my concerns and exchanging ideas how to do it better. I read this email from July 2006 now again and it seems I was right. Citing

"I think there are people, who would implement for example the factorization (you can copy it from eigenmath for example) and other stuff like limits and integration, if the code weren't so
complex (or am I wrong?)."

I was right, except limits (that I wrote myself), all of the above was implemented by someone else in SymPy (Mateusz, but others too, did a stellar job here). It's actually pretty amusing for me to read my other emails from the ginac list, for example this. I basically said the design of SymPy in that email. As one of replies, I was told to consider education in C++. As I also found later, ginac developers get very easily attacking me over nothing. But if one ignores it, I had several interesting discussions with them about a CAS design (we also discussed something later, but I cannot find it now). But anyway, flaming people is something I would never tolerate on a sympy list, because this is exactly what drives people away. And I am not interested in any flame. I just want to get things done. Anyway, I did exactly what I said in those emails on the ginac list with sympy and it seems it worked really well. One remainging piece of the puzzle is speed though. As I watched one Linus talk today, he said "well, maybe 30s is not much for you, but if people are used to do the same thing in 0.1s, trust me, it is way too slow." He's of course 100% right. So we need to speed sympy up drastically if we want to be competitive.

So those were my thoughts recently. So I wrote a SymPy core in pure C over the last weekend, see my emails to the sympy list. I achieved some pretty competitive benchmarks, but there was one "minor" issue -- memory management. I wasn't freeing any memory as I thought I could fix that later easily. And it turned out to be a night mare. I hoped I could use boehm-gc, but it doesn't work with glib (it gives nasty segfaults), that I was using for dictionaries and string management. So I hacked up reference counting just like in Python, but I didn't manage to make it work without segfaults. Segfaults can be debugged, but it's a lot of effort, especially if it fails deeply in some libc function, far away from the code that actually breaks it. Valgrind helps here too, but I want to spend my time on other things than debugging segfaults. Then Robert Bradshaw told me: why not to use Cython? BTW, Kirill was telling me this from the beginning. But I am stubborn, if I believe in something, I want to try it. So I tried and failed. Anyway, I must say I really enjoyed coding in C, it's a very good language.

So then I threw the code away and started from scratch. I think this was the 4th time I wrote the "x+y+x -> 2*x + y" simplification algorithm. For the first time 2 years ago it took me maybe a week to get it right. In the C (3rd time) it took me just maybe an hour. And now (4th time) in Python it was just couple minutes. So it makes me feel good if I can see that I am improving after all. Anyway, so I wrote a core in pure Python, but using all my 2 years experience and a know-how after discussing with Kirill, Mateusz, Fredrik, Pearu and many others how to do things. So our new core, we call it sympyx, is less than 600 lines long, it uses the same architecture as sympy and it is amazingly fast.

Then we reserved couple evenings with Kirill and cythonized it (neither of us has time during the day, I need to work on my thesis and Kirr works). We logged into one machine, setup screen so that we could see each other typing and also type to each other's terminal and started cythonizing. Kirr is very good at it, he submitted couple patches to Cython and he's just much better developer than I am. So he used his approach of rewriting everything directly to Cython, while I used my iterative approach of always trying to satisfy tests. So when he completely broke the core in the middle and I saw on his terminal something like:


In [1]: x+x
[huge stacktrace with Cython errors]

and then I saw the cursor stopped moving, I write to his terminal "haha" and was laughing. Probably a minute later I screwed something up in my terminal and I saw that my cursor wrote haha.
Actually very good moment was that I managed to get all tests run first. So I got hold of his terminal and wrote in his vim session "look at my terminal". And there he saw:


$ py.test
============================= test process starts ==============================
executable:   /usr/bin/python  (2.5.2-final-0)
using py lib: /usr/lib/python2.5/site-packages/py 

test_basic.py[13] .............
test_basic2.py[4] ....

================== tests finished: 17 passed in 0.04 seconds ===================

You know, all tests run, with the core in Cython! And then I saw how my cursor started to move and it was typing "git diff" and "git cherry -v origin" and "git log" and other things, as Kirr was checking that what he sees is correct. But I haven't cdefed the classes, just make it work (there were problems with the __new__ method not yet supported in Cython and other minor things). We both came home from our work and continued, so I continued bringing more and more classes and methods to Cython, but then I got this message from Kirr on my jabber: "all tests pass. Speedup is 3x". So I said to myself -- yes! So he beated me and he did it. I pulled his changes and indeed, everything was cythonized and it was superfast. So his approach provided faster results after all.

We polished things, did couple benchmarks and announced it on sympy and sage-devel lists.

GiNaC is still faster, but not 100x faster, depending on the benchmark usually 2x to 10x faster (but sometimes also slower). There is still a lot of room for both design and technical optimizations, so I think we'll improve for sure. I very strongly believe it is possible to be simple in design, maintainable and fast.

Ubuntu Developer Summit in Prague

2008-05-24T12:48:00.000-07:00

Last weekend I was at FOSSCamp. Since I live in Prague I wanted to go to Ubuntu Developer Summit (UDS) each day, but unfortunately I had some exams, so I only went on Wednesday and Friday.

On Wednesday I first met Lars Wirzenius:

we agreed to go to pub in the evening. Then I did a little work, there was quite a nice view from the window (Prague castle on the horizon):

and I went to the #ubuntu-devel-summit IRC channel and pinged Scott Kitterman, whom I new from the Debian Python Modules Team (DPMT), but didn't know how he looks like. We met and once I knew Scott, it was easy to get around, so he introduced me to Steve Langasek (pronounced Langášek). We agreed to go to pub as well. Steve lives in Portland, OR, where I spent the summer 2005 and Scott is from Baltimore where I spent the summer 2006.
Then I also met Riku Voipio, Martin Böhm, Christian Reis (whom I asked if it's possible to support Debian unstable on Ubuntu Personal Package Archives and he said that it will probably happen, so that's really cool -- I also offered my help with this) and others, so in the end, there were 14 of us going to the pub, so I chose again the same pub as with the FOSSCamp people and it seems it tasted good again:

Notice the svíčková na smetaně above, my favourite meal. Good choice Scott. :)

Ok, that was on Wednesday. On Friday I arrived at around 3pm, looked at the schedule table and noticed that Matthias Klose should be at UDS too, so I started IRC and pinged him. Fortunately, Emilio Pozuelo Monfort, whom I know from DPMT as well, replied first so we met, it was cool and he showed where Matthias is. I am very glad I met him, so we discussed python-central and python-support packages and why we have them both, also with Scott later on.

When I was waiting for Matthias, I sat next to Nicolas Valcárcel, started my laptop and begun looking at some SymPy bugs and Nicolas noticed that and said -- "You are developing SymPy?", I said "Yes.", flattered. And he showed me a bug with plotting and Compiz, so we immediately reported that to pyglet.

In the evening people continued to some kind of a party, but unfortunately, I was already going to some other pub.

Overall, even though I was there for only two afternoons, it was just awesome and I utterly enjoyed meeting all the people I knew from mailinglists and IRC.

Installing a printer in Debian

2008-05-20T08:25:00.000-07:00

I finally bought a new toner to my old Minolta 1250e that I didn't use for a few years (because I was lazy to fix it) and I was surprised how easy it is to install it these days:

1. plugged the USB cable from the printer to my laptop
2. wajig install cupsys foomatic-db-engine
3. sudo ln -s /usr/bin/foomatic-ppdfile /usr/lib/cups/driver
4. On http://localhost:631/ I added the new printer in cupsys, it showed my printer over USB, I selected it and then I chose the correct PPD.

And everything just works. The only nontrivial part is the step 3, that was suggested in /usr/share/doc/foomatic-db-engine/USAGE.gz:


If the printers of the Foomatic database do not appear, check whether the
link to foomatic-ppdfile is in /usr/lib/cups/driver:

lrwxrwxrwx  1 root root 25 Apr 19 18:13 foomatic -> /usr/bin/foomatic-ppdfile

If not, create it manually.

Without it, I didn't see the right PPD in the step 4. Maybe it'd be a nice idea to do this automatically (either when installing foomatic-db-engine or cupsys), or am I missing something?

As to the PPDs, one can get all printer IDs by:


$ foomatic-ppdfile -A

and the PPD for my printer by


$ foomatic-ppdfile -p Minolta-PagePro_1250E

However, it's not necessary in the above howto.

FOSSCamp, Friday

2008-05-16T16:16:00.000-07:00

Since I live in Prague, it's basically compulsory to go to FOSSCamp. Yesterday I went with Lucas to some pubs + sightseeing, today we went in a larger group to this pub:

and we had a couple of good Czech meals with Plzeň beer:

Seems it tasted good:

snapshot.debian.net saved me again

2008-05-07T01:38:00.000-07:00

On one computer I am taking care of, I suddenly started getting:


$ ps2pdf fa_808.ps fa_808.pdf
/usr/bin/ps2pdfwr: line 45: exec: gs: not found

What's wrong?


$ ls /usr/bin/gs
ls: cannot access /usr/bin/gs: No such file or directory
$ wajig find-file /usr/bin/gs
ghostscript: /usr/bin/gs
$ wajig list ghostscript
ii  ghostscript                                             8.61.dfsg.1-1               The GPL Ghostscript PostScript/PDF interpret
ii  ghostscript-x                                           8.61.dfsg.1-1               The GPL Ghostscript PostScript/PDF interpret

That is really weird, the file /usr/bin/gs is simply missing, even though I have the ghostscript package installed. Ok, let's reinstall it:


$ wajig reinstall ghostscript
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libwine-capi libstdc++5 libopenal0a xgnokii libglut3 libcapi20-3
  libggi-target-x lib64gfortran2 libsvg1 lib64gomp1 libggi2 libgii1
  libgii1-target-x libxine1-gnome cups-pdf lib64objc2
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  akregator gs kaddressbook kaddressbook-plugins kalarm kandy kappfinder karm
  kate kcontrol kdebase-bin kdebase-data kdelibs-data kdelibs4c2a
  kdepim-kfile-plugins kdepim-kio-plugins kdepim-kresources kdepim-wizards
  kdesktop kghostview kicker kitchensync kleopatra kmplayer kmplayer-common
  kpersonalizer ksplash libarts1c2a libgnutls26 libgs8 libilmbase6 libkcal2b
  libkdepim1a libkleopatra1 libkmime2 libkonq4 libkpimidentities1 libktnef1
  libopenexr6
Suggested packages:
  kdeaddons-doc-html ntpdate ntp-simple perl-suid egroupware ffmpeg xawtv
  gnutls-bin
The following packages will be REMOVED:
  digikam kde-amusements kde-core kdeaddons kdebase kdebase-kio-plugins kdepim
  kmail kmailcvt kmplayer-plugin knights konq-plugins konqueror
  konqueror-nsplugins korn smb4k
The following NEW packages will be installed:
  gs libgnutls26 libilmbase6 libopenexr6
The following packages will be upgraded:
  akregator ghostscript kaddressbook kaddressbook-plugins kalarm kandy
  kappfinder karm kate kcontrol kdebase-bin kdebase-data kdelibs-data
  kdelibs4c2a kdepim-kfile-plugins kdepim-kio-plugins kdepim-kresources
  kdepim-wizards kdesktop kghostview kicker kitchensync kleopatra kmplayer
  kmplayer-common kpersonalizer ksplash libarts1c2a libgs8 libkcal2b
  libkdepim1a libkleopatra1 libkmime2 libkonq4 libkpimidentities1 libktnef1
36 upgraded, 4 newly installed, 16 to remove and 683 not upgraded.
Need to get 53.1MB of archives.
After this operation, 42.9MB disk space will be freed.
Do you want to continue [Y/n]? n
Abort.

Oops, unstable is broken at the moment. Ok, what now? Well, snapshot.debian.net comes to rescue again. Find "ghostcript", version "8.61.dfsg.1-1" and here we are:


$ wget http://snapshot.debian.net/archive/2008/03/02/debian/pool/main/g/ghostscript/ghostscript_8.61.dfsg.1-1_i386.deb
$ wajig install ./ghostscript_8.61.dfsg.1-1_i386.deb

And all is fine now:


$ ls /usr/bin/gs
/usr/bin/gs

SymPy accepts Google Summer of Code applications

2008-03-25T18:21:00.000-07:00

SymPy is a pure Python library for symbolic mathematics. Last year SymPy had 5 excellent students and this year we are accepting students again.
Why should you apply? And why to SymPy?

Well, let me give you some reasons:

First of all, it's fun. To get some idea, read the GSoC2007 SymPy page, where you can find out what the last year students did and especially read their reports, where they describe their impressions from the summer, how they tackled problems and their overall conclusions.
It's not just about coding, we enjoy the social part too. There is a great community around numpy, scipy, ipython, matplotlib, Sage and similar tools and if you do scientific computing with Python, you gain a lot just being part of it, because you learn new things from the others.
I currently live in Prague (most people say it's a beautiful city, but I actually like Los Angeles, or the Bay Area:), if there are enough interested people, we can make a coding sprint here (plus of course some sightseeing+pubs). Anyone with a good commit history is welcome to stay at my apartment. :)
You earn $4500, some of which I suggest to spend on travelling to conferences/workshops, here are some tips: SciPy2008 (see also SciPy2007), EuroSciPy2008, Sage Days (you can read my impressions from SD6 and SD8), watch the numpy/scipy mailinglists for announcement of other meetings.

Read also the current status and motivation of SymPy and it's relation to Sage. If you want to apply, all the necessary information is on our wiki page.

Nevertheless, if you decide SymPy is not for you, but still you'd like to do GSoC project in a similar area, there are other good options too - one is SciPy/NumPy, the other is Sage. Unfortunately Sage was not accepted as a mentorship organization, but it has several good projects too, some of which you can do for example under the umbrella of the Python Software Foundation.

One of them is improving the Sage notebook. If you've never seen that - download Sage, start it (./sage), type "notebook()" and a nice Mathematica like notebook will popup in the browser. It allows collaborative editing ala Google Docs and many other things. If you'd like to work on it, reply to the email on sage-devel.

Sage Days 8

2008-03-06T14:50:00.000-08:00

Between February 29 and March 4, 2008 I attended the Sage Days 8, hosted at the Enthought headquarters in Austin, Texas. This was my 5th time in the USA and it was a marvelous experience, as with all my visits in the states.

As usual, I had some adventures in Atlanta, that interested readers can find at the end of this post. Anyway, on the Austin's airport I met Peter and his wife Crystal, Fernando, Benjamin, Jarrod, Eric and Clement. We went to have a dinner and then me and Clement were staying at Peter's house:

You can see the neighbor's cat and Peter's dog Trinity behind the window. The next day we went to Enthought, that was providing us with a breakfast and a lunch each day - and it was delicious. After the breakfast, we gathered in the room and introduced ourselves. Enthought rents 3/4 of the 21th floor in the Bank of America building, so when I looked left I saw:

When I looked behind I saw:

and in front of me, I saw all the participants (I took photos of all participants together with names). As you can see, there were really good people in there, like Travis (creator of NumPy), William (main author of Sage), Eric (CEO of Enthought), Fernando (author of IPython), Jarrod (the release manager of SciPy), Michael (the release manager of Sage) etc. See also the Fernando's welcome speech and the video of each of us introducting himself.

The views from the windows are terrific. I enjoyed working on each of the 4 sides of the skyscraper with completely different scenery, or when the sun is going down, that's also very cool.

We spent the whole Friday doing presentations, some of which you can find here. Then we went to Eric's house to have a big dinner together.

On Saturday, Sunday and Monday we were all hacking on many different things. I joined Fernando, Benjamin, Brian and Stefan on ipython1, Travis was implementing a new type (gmp integer) in NumPy, William wrote a manipulate command in Sage, Eric did the same in Traits, Gary and Michael implemented parallel testing of Sage, ...

On Tuesday we had final status reports and people left in the afternoon. In the evening we went with Clement to have a dinner and then we visited some bars on the 6th street, having a beer in each.

On Wednesday I visited John and Roy from the Computational Fluid Dynamics Lab at the University of Texas, Austin, who wrote the libMesh library, that I extensively used and also created a Debian package of. It was very influential to see the libMesh "from behind", also John and Roy are cool people (not mentioning the Debian tradition of having good relations with upstream:). Then I visited some professors at the same campus, after which I went into the Capitol and then I took the bus to the Barton Creek Square Mall to buy some ipods and jeans, so that I can say I have jeans from Texas. BTW, the ipod works excellent in Debian - I plugged it in and it just shows on my Gnome desktop. It's true that naively dragging mp3 files on it didn't make it play, but these instructions made it work.

On Thursday I fixed the remaining release blockers in SymPy and made a new release. In the evening, I am going to meet Aswin, he also uses SciPy and also is a friend of Kumar, who is now maintaining python-numpy and python-scipy Debian packages with me (Kumar also knows Prabhu, the author of Mayavi2 hosted at Enthought, so it's all connected).

Anyway, the whole workshop was an excellent experience for me. I learned a lot of new things and being able to speak with people who wrote tools that I use almost everyday is important. We also extensively discussed the future of all the projects (Sage, SciPy, NumPy, IPython, Cython, SymPy). See my summarizing email to the SymPy mailinglist.

Another thing, that I find very interesting is that Microsoft is financing the windows port of Sage, that will make basically anything that uses Python/Cython/C/Fortran very easy to install on windows (just a spkg package in sage). I find it really cool that MS is not only supporting but even financing a truly opensource project.

Finally the promised adventure in Atlanta: we took off the Prague airport on February 28th with a 2 hours delay (due to some paperwork as we were told by the captain). As I had 3 hours in Atlanta for the connection to Austin and I had to go through immigration, it was clear that I'll miss it. But I was not surprised, last time I was flying through Atlanta, they canceled my flight to LA completely. We arrived in Atlanta an hour and a half before my departure, then I was waiting for about an hour at immigration, it was incredibly slow. When I had around 20 min to departure, I had to ask people standing in front of me if they let me in, they were very nice and did. I was leaving immigration 10 min to my departure, then I was running to get my luggage and myself through customs and screening, it was 5 min to my departure when I ran down to the display with departure times. Then I was sprinting like hell to the terminal D to only see the clerk doing some final paperwork with all the people already boarded and the jetway door shut. After a little persuading he let me in too, fortunately there was still one seat left, so I made it. You can imagine my pleasant surprise in Austin when I discovered, that my luggage made it too, considering that I handed it to the Atlanta's airport personnel exactly 10 min prior the departure.

XFS is 20x slower than ext3 (with default settings)

2008-02-26T17:00:00.000-08:00

Is XFS that bad? Well, at least with default settings, XFS on Debian seems to be blown away by ext3 completely in terms of speed. I don't mind 1.5x slowdown, maybe even 2x, but 20x is a show stopper. I am already using ext3 for any pbuilder builds, because it's a difference to wait for 30s with XFS, compared to 3s with ext3 to extract the base image. And I'll probably switch to ext3 completely, unless someone finds a way how to fix this.

I recently got burned by this when running Sage on my computer, because it compiles a lot of Python files when started for the first time. Normally it should take roughly 15s, but instead it took 6 minutes on my comp and then it triggered a so far undiscovered bug in Sage, that I reported.

Michael Abshoff, the release manager of Sage, suggested that something is FUBAR (Fucked Up Beyond Any Recognition) on my shiny Debian amd64 sid system running on Intel Core Quad, so I said no way, because I really care about this machine, as I use it for larger finite elements calculations and other stuff (like compiling huge deb packages in parallel, like paraview).

So I offered a bet, that I give him an access to this compter, he finds the problem and if it's a problem in my Debian configuration, I'll write to this blog that I am lame, while if it's a problem in Sage, he will write to his blog that he is lame. And I was smiling to myself, how good I am and that I will have some fun too reading planet.sagemath.org with the top post from Michael saying that he is lame.

But then I remembered my old struggle with cowbuilder and XFS and I stopped smiling. See e.g. this wiki I created half a year ago. Something is FUBAR with XFS and Debian. I also asked on the Czech server Root, that is famous for having a lot of experts willing to share their knowledge, and it was quickly revealed, that the problem is with the "nobarrier" option of XFS (my post is here, but it's in Czech).

First, on that amd64 machine, the above problem was fixed after issuing this command:


mount -o remount,rw,nobarrier /dev/sda3 /home/

(notice the "nobarrier" option). You can read some background behind this on the lkml list. Unfortunately, I also have my laptop, and there I already use this "nobarrier" option, and it doesn't help at all. I just created a new ext3 partition and verified that on my laptop, ext3 is around 10x faster than XFS with nobarrier (that was supposed to fix this). I use the latest 2.6.24 kernel from unstable on both.

Time to move from XFS to ext3 on my laptop? Seems like that. I'll leave XFS on the other machine, because I know some other peole have good experience with XFS and the "nobarrier" option seems to fix the problem there.

But as to the bet, yeah, I am lame and I should still learn a lot from Michael. :)

SymPy/sympycore (pure Python) up to 5x faster than Maxima (future of Sage.calculus?)

2008-01-03T07:40:00.000-08:00

According to this test, sympycore is from 2.5x to 5x faster than Maxima. This is an absolutely fantastic result and also a perfect certificate for Python in scientific computing. Considering that we compare pure Python to LISP.

Ok, this made us excited, so we dugg deeper and ran more benchmarks. But first, let me say a few general remarks. I want a fast CAS (Computer Algebra System) in Python. General CAS, that people use, that is useful, that is easily extensible (!), that is not missing anything, that is comparable to Mathematica and Maple -- and most importantly -- I want it now and I don't care about 30 years horizons (I won't be able to do any serious programming in 30 years anyway). All right. How to do that? Well, many people tried... And failed. The only opensource CAS system, that has any chance of becoming the opensource CAS, in my own opinion, is Sage. You can read more about my impressions form Sage here. I am actually only interested in mathematical physics, so basically Sage.calculus. Currently Sage uses Maxima, because Maxima is old, proven, working system and it's reasonably fast and quite reliable, but written in LISP. Some people like LISP. I don't and I find it extremely difficult to extend Maxima. Also even though Maxima is in LISP, it uses it's own language for interacting with the user (well, that's not the way). I like python, so I want to use Python. Sage has written Python wrappers to Maxima, so Sage can do almost everything that Maxima can, plus many other things. Now. But the Sage.calculus has issues.

First, I don't know how to extend the wrappers with some new things, see my post in the sage-devel for details, it's almost 2 months old with no reaction, which shows that it's a difficult issue (or nonsense:)).

And second, it's slow. For some examples that Sage users have found out, even SymPy, as it is now, is 7x faster than Sage and sympycore 23x faster and with the recent speed improvements 40x faster than Sage.

So let's improve Sage.calculus. How? Well, no one knows for sure, but
I believe in my original idea of pure Python CAS (SymPy), possibly with some parts rewritten in C. Fortunately, quite a lot of us believe that this is the way.

What is this sympycore thing? In sympy, we wanted to have something now, instead of tomorrow, so we were adding a lot of features, not looking too much on speed. But then Pearu Peterson came and said, guys, we need speed too. So he rewrote the core (resulting in 10x to 100x speedup) and we moved to the new core. But first, the speed isn't sufficient, and second it destabilized SymPy a lot (there are still some problems with caching and assumptions half a year later). So with the next package of speed improvements, we decided to either port them to the current sympy, or wait until the new core stabilizes enough. So the new new core is called sympycore now, currently it only has the very basic arithmetics (and derivatives and simple integrals), but it's very fast. It's mainly done by Pearu. But for example the latest speed improvement using sexpressions was invented by Fredrik Johansson, another SymPy developer and the author of mpmath.

OK, let's go back to the benchmarks. First thing we realized is that Pearu was using CLISP 2.41 (2006-10-13) and compiled Maxima by hand in the above timings, but when I tried Maxima in Debian (which is compiled with GNU Common Lisp (GCL) GCL 2.6.8), I got different results, Maxima did beat sympycore.

SymPyCore:


In [5]: %time e=((x+y+z)**100).expand()
CPU times: user 0.57 s, sys: 0.00 s, total: 0.57 s
Wall time: 0.57

In [6]: %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.25 s, sys: 0.00 s, total: 0.25 s
Wall time: 0.25

Maxima:


(%i7) t0:elapsed_real_time ()$ expand ((x+y+z)^100)$ elapsed_real_time ()-t0;
(%o9)                                0.41
(%i16) t0:elapsed_real_time ()$ expand ((x + y+z)^20*(x+z)^19)$ elapsed_real_time ()-t0;
(%o18)                         0.080000000000005

So when expanding, Maxima is comparable to sympycore (0.41 vs 0.57), but for general arithmetics, Maxima is 3.5x faster. We also compared GiNaC (resp. swiginac):


>>> %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.03 s, sys: 0.00 s, total: 0.03 s
Wall time: 0.03

Then we compared just the (x+y+z)**200:


sympycore:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 1.80 s, sys: 0.06 s, total: 1.86 s
Wall time: 1.92
swiginac:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 0.52 s, sys: 0.02 s, total: 0.53 s
maxima:
(%i41) t0:elapsed_real_time ()$ expand ((x + y+z)^200)$ elapsed_real_time ()-t0;
(%o43)                         2.220000000000027

Where GiNaC still wins, but sympycore beats Maxima, but the timings really depend on the algorithm used, sympycore uses Millers algorithm which is the most efficient.

So then we tried a fair comparison: compare expanding x * y where x and y are expanded powers (to make more terms):


 sympycore:
 >>> from sympy import *
 >>> x,y,z=map(Symbol,'xyz')
 >>> xx=((x+y+z)**20).expand()
 >>> yy=((x+y+z)**21).expand()
 >>> %time e=(xx*yy).expand()
 CPU times: user 2.21 s, sys: 0.10 s, total: 2.32 s
 Wall time: 2.31
 swiginac:
 >>> xx=((x+y+z)**20).expand()
 >>> yy=((x+y+z)**21).expand()
 >>> %time e=(xx*yy).expand()
 CPU times: user 0.30 s, sys: 0.00 s, total: 0.30 s
 Wall time: 0.30
 maxima:
 (%i44) xx:expand((x+y+z)^20)$
 (%i45) yy:expand((x+y+z)^21)$
 (%i46) t0:elapsed_real_time ()$ expand (xx*yy)$ elapsed_real_time ()-t0;
 (%o48)                         0.57999999999993

So, sympycore is 7x slower than swiginac and 3x slower than maxima. We are still using pure Python, so that's very promising.

When using sexpr functions directly then 3*(a*x+..) is 4-5x faster than Maxima in Debian/Ubuntu. So, the headline of this post is justified. :)

Conclusion

Let's build the car. Sage has the most features and it is the most complete car. It has issues, some wheels need to be improved (Sage.calculus). Let's change them then. Maybe SymPy could be the new wheel, maybe not, we'll see. SymPy is quite a reasonable car for calculus (it has plotting, it has exports to latex, nice, simple but powerfull command line with ipython and all those bells and whistles and it can also be used as a regular python library). But it also has issues, one wheel should be improved. That's the sympycore project.

All those smaller and smaller wheels show, that this is indeed the way to go, but very important thing is to put them back in the car. I.e. sympycore back to sympy and sympy back to Sage and integrate them well. While also leaving them as separate modules, so that users, that only need one particular wheel, can use them.

R. = QQ[] what is that?

2008-01-01T12:47:00.000-08:00

While playing with Jaap's wish, I finally got fedup and decided to study what the funny syntax "R.<a,b,c> = QQ[]" really means. So I did:


sage: R. = QQ[]
sage: preparse("R. = QQ[]")
"R = QQ['a, b, c']; (a, b, c,) = R._first_ngens(Integer(3))"

Now everything is crystal clear! Let's study what QQ is:


sage: QQ?   
Type:  RationalField
Base Class: 
String Form: Rational Field
Namespace: Interactive
Docstring:
    
        The class class{RationalField} represents the field Q of
        rational numbers.

Cool, why not. So what is the _first_ngens method doing? Let's find out:


sage: R._first_ngens?
Type:  builtin_function_or_method
Base Class: 
String Form: 
Namespace: Interactive

Hm, not really useful. Let's push harder:


sage: R._first_ngens??
Type:  builtin_function_or_method
Base Class: 
String Form: 
Namespace: Interactive
Source:
    def _first_ngens(self, n):
        v = self.gens()
        return v[:n]

So the equivalent code is this:


sage: R.gens()
(a, b, c)
sage: R.gens()[:3]
(a, b, c)

Cool, why not. So what is the R.gens() doing?


sage: R.gens? 
Type:  builtin_function_or_method
Base Class: 
String Form: 
Namespace: Interactive
Docstring:
    
            Return the tuple of variables in self.
    
            EXAMPLES:
                sage: P. = QQ[]
                sage: P.gens()
                (x, y, z)
    
                sage: P = MPolynomialRing(QQ,10,'x')
                sage: P.gens()
                (x0, x1, x2, x3, x4, x5, x6, x7, x8, x9)
    
                sage: P. = MPolynomialRing(QQ,2) # weird names
                sage: P.gens()
                (SAGE, SINGULAR)

Ah, that's the answer we want. :)

Some timings of startups

2007-12-25T16:53:00.000-08:00

I did some tests (I always start the program and then immediatelly hit ctrl-D):

$ time ipython
[...]

real 0m0.305s
user 0m0.112s
sys 0m0.024s

$ time bin/isympy
[...]

real 0m0.535s
user 0m0.200s
sys 0m0.040s

$ time ./sage
[...]

real 0m1.398s
user 0m0.916s
sys 0m0.208s

I did that repeatedly, so this is the usual time I get on my laptop. I think ipython is quite slow, because:

In [1]: %time import sympy
CPU times: user 0.08 s, sys: 0.01 s, total: 0.09 s
Wall time: 0.09

or if I import it in the python interpreter, it's immediate.

The 0.3s is acceptable, 0.5s is quite a lot for me, and 1.4s is a lot. I am maybe too demanding, but it just annoys me to wait for 1.4s on the import time. I just get some idea to try, so I fire up Sage or SymPy and try it. I don't want to wait for 1.4s.

This can be fixed, one just needs to find modules that slow things down, and late import them, or just fix them in some way. And usually, one does it several times -- I think Sage did that twice, SymPy also I think did that twice already.

People still (and will) use windows

2007-12-22T09:28:00.000-08:00

I sometimes look at the download statistics of SymPy and the funny thing is that people download more the Windows installer, rather than the multiplatform source tarball. Part of this can be due to the fact, that SymPy is in Debian, Ubuntu and Sage, but still.

So here is how to make a windows installer in Debian:


$ ./setup.py bdist_wininst
[...]

and a file dist/sympy-0.5.9-hg.win32.exe is created. I only have regular python tools installed. So we just upload the exe to our site and that's it -- kind of scary, since I don't have means to test it.

But one time I tried that on one of brother's computers with windows and not only it worked, but also the 3D plotting using pyglet worked out of the box! If you are curious how it looks like:

So it has it's advantages to develop pure Python programs - they really run everywhere.

I think installing things like Python and playing with SymPy on windows must be pain (maybe people just download SymPy for windows and then run away with disgust). But just the fact that they try means, that people doing science do use windows a lot. And I don't think this is going to change any time soon. I also fully agree with Michael Abshoff's recent post about this issue.

Mérida - wrap up

2007-12-05T01:20:00.000-08:00

From November 28 till December 2, 2007, I attended a Debian QA and release teams work session in Extremadura, which is an autonomous community of western Spain, that managed to install Debian on 90000 computers in every school (technically a Debian based distribution called gnuLinEx) and Junta de Extremadura also sponsored this meeting.

I took photos of all participants, see my first, second and third posts. When I arrived at the Madrid airport where we first met, I set myself a goal to remember all names and faces, so I used my blog to help me and I think I succeeded in the end. :)

See also our wiki page that we used prior and during the meeting.

So what did we do besides throwing candies?

Lucas Nussbaum will send a summary email soon about the meeting, so I'll just speak for myself:

I worked with Gonéri on svnbuildstat, that is a service for building packages and show statistics about lintian/linda/piuparts checks. It for example contains all packages of the Debian Python Modules Team (that I am a member of) and many others. We discussed and started to work on how to create robust buildbots, that can be installed as a regular Debian package with zero (if possible) configuration, so that many people can just install them without pain, thus providing a huge scalability to the project.

We wrote a preliminary patch to pbuilder for killing the build if it exceeds given memory/disk usage. I had to learn the internals of pbuilder and I lost quite some time squashing some stupid bug I caused while writing the patch.

I spent most of my time with svnbuildstat, mostly learning and discussing things. This will be important for the future, but to also have some real results, I also fixed some packages I comaintain:

Together with Kumar Appaiah we fixed the python-numpy package and I had it uploaded, then I learned how to work with quilt instead of dpatch to handle patches in Debian packages, thanks to Holger's webpage, that contains a nice tutorial. Then I switched from dpatch to quilt in python-scipy and backported a patch from upstream svn to fix a segfault bug and had the package uploaded.

Then I finished the Cython package and had it uploaded. Cython is a marvelous package to speed up Python programs and interface C/C++ programs. I greatly recommend to try. If you don't like it, you can try some of at least 10 other ways to wrap C code in Python. I also used quilt in there to backport a patch from the upstream Mercurial repository to implement parsing @classmethods. Quilt is really a pleasure to work with.

Impressions from the meeting

I've been using Debian since 2001 as my only operating system on all of my computers, so I am not a complete beginner. But it never occured to me I could get involved in Debian more than a user and an occasional bug reporter. What a mistake.

I started packaging new things and fixing packages that I need for my work and that didn't work. This got me involved quite a bit in Debian. But in Mérida it was the first time I could dring a beer (well, especially wine) with Debian Developers and I found out they are really cool people. They are all very skilled. Also something, that I love about Debian, is that the people involved in it share two common features, that are very important for them - respect to democracy and personal freedom. When I think about it, those are probably the first two items on my presonal list of values.

Of course, everytime there is a group of 1000+ people, there are good and bad people, more and less skilled, but important is the overall atmosphere - and that is as I described. I think Debian is truly unique. There is Gentoo, that has maybe 40 (?) active developers. There is Ubuntu, that has maybe 100 (?) developers, but it's basically a comercial distribution and there is not so many interesting work for non employees of Canonical. There is opensuse and fedora, where I am not sure about the numbers. The atmosphere in Debian can change in the future, one never knows, but as of the end of 2007, I think it's very cool to get involved.

Maybe it's not for everyone, but it's the right place for me.

Mérida - remaining photo

2007-12-04T13:45:00.000-08:00

This post is dedicated to dato (blog), whose picture I forgot to take (thanks Cyril Brulebois for taking this one).

From left to right: Gonéri Le Bouder, me, Lars Wirzenius, Holger Levsen and Adeodato "dato" Simó!

Merida

2007-12-01T04:36:00.000-08:00

We spent the whole Friday hacking, Lucas will send a summary email soon. More people joined.

Ana Guerrero:

Amaya Rodrigo:

Mark Purcell:

Kilian Krause and César Gómez Martín

and Holger Levsen with red hair:

On Saturday we were again working and in the late afternoon we took a walk in Mérida, visited the famous ancient Roman monuments. Now it's 3:30am and we will soon go to Madrid and back home.

Debian meeting in Merida, Spain

2007-11-29T03:01:00.000-08:00

Right now, some Debian Developers (and also not yet Developers, like me:), are on
the work sessions in Extremadura, I am on the QA and release teams meeting.

We started in the morning with presentations (see also the schedule). Any comments and suggestions welcomed, please add comments below the post.

Lucas Nussbaum presenting:

Most of us:

And in details, names from left to right. Cyril Brulebois, Gonéri Le Bouder:

Luk Claes, Marc 'HE' Brockschmidt, Jörg Jaspert, Lars Wirzenius:

Fabio Tranchitella, Bernd Zeimetz, Mario Iseli, Luk Claes:

Filippo Giunchedi, Stefano Zacchiroli, Tzafrir Cohen, Simon Richter, Faidon Liambotis:

And again, so that Faidon is visible:

How to connect to the internet using T-Mobile and bluetooth

2007-11-28T04:09:00.000-08:00

I have a laptop with Debian, cell phone (Nokia N70) with bluetooth and this howto describes how to connect to the internet.

Connect USB bluetooth dongle to the laptop and check it:


$ hciconfig
hci0:   Type: USB
    BD Address: 00:02:72:D2:23:12 ACL MTU: 310:10 SCO MTU: 64:8
    UP RUNNING PSCAN ISCAN
    RX bytes:686 acl:0 sco:0 events:22 errors:0
    TX bytes:337 acl:0 sco:0 commands:21 errors:0

Scan for the phone:


$ hcitool scan
Scanning ...
    00:19:79:86:EB:BC       Nokia N70

Create a serial device for communicating with the modem in the phone:


$ rfcomm connect /dev/rfcomm0 00:19:79:86:EB:BC 3
Connected /dev/rfcomm0 to 00:19:79:86:EB:BC on channel 3
Press CTRL-C for hangup

(A dialog will popup on the phone asking me to allow the laptop to connect.)

Connect using wvdial:


$ wvdial
--> WvDial: Internet dialer version 1.56
--> Cannot get information for serial port.
--> Initializing modem.
--> Sending: ATZ
ATZ
OK
--> Sending: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
OK
--> Modem initialized.
--> Sending: ATDT*99***1#
--> Waiting for carrier.
ATDT*99***1#
CONNECT
~[7f]}#@!}!} } }2}#}$@#}!}$}%\}"}&} }*} } g}%~
--> Carrier detected.  Starting PPP immediately.
--> Starting pppd at Wed Nov 28 12:58:55 2007
--> Pid of pppd: 4378
--> Using interface ppp0
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> local  IP address 172.24.171.97
--> pppd: �[08][06][08]�[10][06][08]
--> remote IP address 10.6.6.6
--> pppd: �[08][06][08]�[10][06][08]
--> primary   DNS address 62.141.0.2
--> pppd: �[08][06][08]�[10][06][08]
--> secondary DNS address 213.162.65.1
--> pppd: �[08][06][08]�[10][06][08]

And that's it, I am connected.

Prerequisities


$ wajig install wvdial bluez-utils

And setup the /etc/wvdial.conf file:


$ cat /etc/wvdial.conf
[Dialer Defaults]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Modem Type = USB Modem
Baud = 460800
New PPPD = yes
Modem = /dev/rfcomm0
ISDN = 0
Phone = *99***1#
Password = doesnt_matter
Username = doesnt_matter
Stupid Mode = 1

snapshot.debian.net rocks

2007-11-17T15:46:00.000-08:00

Today during upgrade I got the gnome packages broken:


$ wajig install nautilus
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.

Since you only requested a single operation it is extremely likely that
the package is simply not installable and a bug report against
that package should be filed.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
  nautilus: Depends: libexempi2 but it is not installable
E: Broken packages
$

I tried to download the source package of libexempi2, no luck (doesn't exist). Then I remembered I've heard about snapshot.debian.net, so I tried that and added:


deb http://snapshot.debian.net/archive pool exempi

to my /etc/apt/sources.list and voilà - it works!


$ wajig install nautilus
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libstdc++5 libpixman-1-0 libglut3 libsqlite0 lib64gfortran2 libqt4-sql
  libntfs-3g4 libkdcraw1
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  fam libexempi2 libtrackerclient0 nautilus-cd-burner
Suggested packages:
  tracker
The following NEW packages will be installed:
  fam libexempi2 libtrackerclient0 nautilus nautilus-cd-burner
0 upgraded, 5 newly installed, 0 to remove and 46 not upgraded.
Need to get 1583kB of archives.
After unpacking 5632kB of additional disk space will be used.
Do you want to continue [Y/n]? 
WARNING: The following packages cannot be authenticated!
  libexempi2
Install these packages without verification [y/N]? y
Get:1 http://ftp.cz.debian.org sid/main fam 2.7.0-13 [69.0kB]
Get:2 http://ftp.cz.debian.org sid/main libtrackerclient0 0.6.3-3 [41.3kB]
Get:3 http://ftp.cz.debian.org sid/main nautilus 2.20.0-1 [639kB]              
Get:4 http://snapshot.debian.net pool/exempi libexempi2 1.99.4-1 [290kB]
Get:5 http://ftp.cz.debian.org sid/main nautilus-cd-burner 2.20.0-1 [544kB]
Fetched 1583kB in 7s (199kB/s)                                                 
Selecting previously deselected package fam.
(Reading database ... 210515 files and directories currently installed.)
Unpacking fam (from .../archives/fam_2.7.0-13_i386.deb) ...
Selecting previously deselected package libexempi2.
Unpacking libexempi2 (from .../libexempi2_1.99.4-1_i386.deb) ...
Selecting previously deselected package libtrackerclient0.
Unpacking libtrackerclient0 (from .../libtrackerclient0_0.6.3-3_i386.deb) ...
Selecting previously deselected package nautilus.
Unpacking nautilus (from .../nautilus_2.20.0-1_i386.deb) ...
dpkg: warning - unable to delete old directory `/usr/share/mime/application': Directory not empty
Selecting previously deselected package nautilus-cd-burner.
Unpacking nautilus-cd-burner (from .../nautilus-cd-burner_2.20.0-1_i386.deb) ...
Setting up fam (2.7.0-13) ...
Starting file alteration monitor: FAM.
Setting up libexempi2 (1.99.4-1) ...
Setting up libtrackerclient0 (0.6.3-3) ...
Setting up nautilus (2.20.0-1) ...
Setting up nautilus-cd-burner (2.20.0-1) ...

SAGE Days 6

2007-11-14T15:40:00.000-08:00

From November 9 till November 15 I attended SAGE Days 6 in Bristol, UK. It was a conference and a coding sprint for the SAGE project, which wants to build something comparable to Mathematica, Maple, Matlab and Magma built only from open source components without reinventing the wheel. Sage also provides excellent novel implementations for many mathematical algorithms (not found in Mathematica/Maple etc.).

I was invited to give a talk about SymPy, which is a Python library for symbolic mathematics, that we are doing.

Here are my notes from each day:

Saturday: one, two
Sunday: one, two
Monday, Tuesday, Wednesday: one

Impressions

Very positive. I started SymPy two years ago, because I wanted to play with symbolic mathematics in Python, see for example my presentation I gave at this conference for details. SAGE at that time was just able to do some mathematics things, but it was very weak in calculus, which is the SymPy's main domain. This has changed last half a year, when SAGE people managed to wrap Maxima in Python (which I thought to be completely impossible), so I started to follow SAGE development more closely. After the SD6, I must say I became very excited about the project.

One problem with mailinglists, IRC and other online interaction is that it's very difficult for me to get an impression about the people and the project. Being able to meet the developers and discuss with them face to face gives me the impression very quickly and very accurately.

By far the biggest guarantee, why it is worthy for me to contribute to SAGE, is the project leader, William Stein. He is very rational and pragmatic (I like these two properties) and after many discussions with him, I came to realize that he has basically identical views on the important things as I do and were I on his place, I would do the same decisions as he did and does. That's very nice, because I can concentrate my energy on things that I like to improve and don't have to worry about other things, because I know he will do it right.

The other SAGE developers are experts and with similar attitude as William has. It's enthusiastic to be among people who make things happen. For example one of the authors of Cython, Robert, implemented during SD6 a very nice HTML output, that shows Cython code, with colors according to how many Python API calls are called on that particular line, and by clicking that line it shows the corresponding C code.

SAGE project has a high aim and it stricly goes for it, without looking too much to the right or left, and that's how it should be. And it does produce a lot of very useful and high quality stuff along the way, for example Cython (probably the best wrapper for C/C++ things now, only pypy could possibly beat it, but that's still more a research project) or the SAGE notebook, which looks like a Mathematica notebook, but better and in a browser (together with a revision history, sharing, SSL encryption, etc.).

The only little problem is that currently SAGE developers are all mathematicians and as is well-known, mathematicians looks at mathematic from a very different prospective than physicists. :) And so I need calculus, advanced calculus and only when this is working, and working well, I can build on it some more advanced features. SAGE currently goes a little the other way - it has a lot of advanced features, from number theory, modular forms, elliptic curves, etc., but the basic calculus still needs a lot of improvements. SAGE wraps Maxima, because Maxima is quite fast, very well tested, so it works well. It's difficult to extend and written in LISP and that's very bad. That's where SymPy could help - it's in Python, very easy to extend, but currently slower than Maxima (rewriting parts of SymPy using Cython, or even C directly, will make it faster, hopefully as fast as Maxima or faster).

It's not yet in Debian, but SAGE people are working on it. It's not an easy task unfortunately.

Conclusion

SAGE is a very promising young project and I think it will succeed to provide an open source alternative to Maple, Matlab, Mathematica and Magma.

Monday, Tuesday, Wednesday

2007-11-14T14:47:00.001-08:00

Monday

On Monday we, as usual, had a breakfast in the Marriott Hotel:

and went to the Heilbronn Institute:

that is right across the highest point in Bristol:

We had a presentation by Gregory Bard:

Then we had a discussion what things should be done in the following coding sprints. Then we coded and in the afternoon I had a presentation "SymPy: A Python library for symbolic mathematics".

Tuesday

In the morning Michael Abshoff gave a presentation about valgrinding a Python/Cython/C/C++ library application like Sage (which adds complexity and its own set of problems because of python). We spent the whole day coding, in the afternoon we did a quick session of lightning talks and in the evening we went to a pub, then we again coded till 1am, some even later.

Wednesday

In the morning we did a little coding and then had a wrap-up session:

Then William had a presentation "SAGE for number theorists" at the Heilbronn Seminar, after which most of the people went to pub, but I with Jaap Spies went to the Old City, as tourists. Then I visited a bookstore, read some books and then we had a last coding evening:

Sunday afternoon

2007-11-11T14:25:00.001-08:00

After a lunch we had a presentation by Clement Pernet about Fast Exact Linear Algebra, then David Loeffler talked about Computing Automorphic Forms for Unitary Groups using Sage:

and then Bill Hart gave a presentation about Algebraic Number Theory with Flint:

In the evening we went to an Indian restaurant and then we had a panel discussion
"The Future of Open Source Mathematical Software", moderated by John Cremona with panelists William Stein, Bill Allombert, Michael Abshoff, Dan Bernstein and me. Recording should be available after the conference.

In the evening, we went to the usual code sprint.

Sunday morning

2007-11-11T02:17:00.000-08:00

After fighting with the famous British separated hot and cold water taps in the morning, I had the traditional bacon and eggs:

And it was delicious. Then we had the first presentation by Michael Brickenstein (see also the schedule):

After a break, we witnessed a marvelous presentation by Paul Zimmerman about MPFR, used by default by gcc and gfortran:

And we went to lunch.

The rest of Saturday

2007-11-10T15:53:00.000-08:00

We went to a common lunch at the institute, after which another 3 presentations followed. Robert Bradshaw on Cython, James Davenport on simplification in CAS and Dan Bernstein about elliptic curves. (Most people here are mathematicians, for example working on modular forms, cryptography etc.)

Then we had a conference dinner together, on which we discussed a lot of stuff - GPL2 vs GPL3 vs BSD pros and cons and problems and how to license SAGE and SymPy and whether "translating" something from C++ to Python is a derived work or not, I also asked William, the main author of SAGE sitting on my left hand side, how SAGE has started and about his future plans with it. Basically, we exchanged our experiences with working on an opensource project and how to motivate people to work on it, and how to do and especially not to do things. On my right hand side was Bill Allombert, one of the two authors of PARI/GP and also a Debian Developer, so we signed each other's gpg keys (that we use in Debian) earlier this afternoon. It's awesome, I am wearing my Debian t-shirt today, but I didn't expect to meet any Debian people here.

After a dinner we went to the math institute to have a coding sprint, some pictures using my cell phone from the sprint:

During the dinner and especially during the sprint I borrowed William's camera and recorded short interviews with conference participants. I am going to do more interviews tomorrow. I'll put the link to them later, when William puts them on the internet after the conference.

Downloading images from Nokia N70 over bluetooth in GNOME

2007-11-10T03:31:00.000-08:00

is actually very easy. I connect the bluetooth dongle to the USB port on my laptop and:


$ wajig install gnome-bluetooth
$ gnome-obex-server
conn_request:   bdaddr 00:19:79:86:EB:BC
conn_complete:  status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '09112007196.jpg' Length 297132
** Message: Saving to '/home/ondra/Desktop/Downloads/09112007196.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC
conn_request:   bdaddr 00:19:79:86:EB:BC
conn_complete:  status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '10112007197.jpg' Length 289214
** Message: Saving to '/home/ondra/Desktop/Downloads/10112007197.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC
conn_request:   bdaddr 00:19:79:86:EB:BC
conn_complete:  status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '10112007198.jpg' Length 287449
** Message: Saving to '/home/ondra/Desktop/Downloads/10112007198.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC

And then I just click on the image on my phone and say "send over bluetooth", it finds my computer, I say yes, and that's it. It ends up in my Downloads folder on the desktop.

Then I just upload the images to my blog, see an example how it looks like.

SAGE Days 6 - Friday and Saturday morning

2007-11-10T03:09:00.000-08:00

Right now I am at the SAGE Days 6 conference.

On Friday, I took the plane

from Prague to Bristol, where we accomodated in the Marriott Hotel:

And went to have a dinner together in a Thai restaurant. On Staturday morning we went to our first session

to the math department:

where William:

and Martin:

gave an introductory talk about SAGE.

Mercurial vs git for managing Debian packages

2007-10-29T15:53:00.000-07:00

I was used to managing all my packages in a Subversion repository, but now when we moved from Subversion to Mercurial in SymPy, I somehow prefer Mercurial for all my work now. :) We chose Mercurial, because I found it a little more polished than git, also the commands are the same as in Subversion.

I like the git-buildpackage, that it manages everything in one repository and that's it, and also that it's written in Python. One would expect that when Mercurial is written in Python, that hg-buildpackage would be too, but it's written in Haskell, so I cannot easily fix it. I filed a wishlist against the hg-buildpackage, whether it is a good idea to provide the same interface as git-buildpackage.

So for now I am going to try git for packaging things in Debian.