Tuesday, December 25, 2007

Some timings of startups

I did some tests (I always start the program and then immediatelly hit ctrl-D):

$ time ipython
[...]

real 0m0.305s
user 0m0.112s
sys 0m0.024s

$ time bin/isympy
[...]

real 0m0.535s
user 0m0.200s
sys 0m0.040s

$ time ./sage
[...]

real 0m1.398s
user 0m0.916s
sys 0m0.208s

I did that repeatedly, so this is the usual time I get on my laptop. I think ipython is quite slow, because:

In [1]: %time import sympy
CPU times: user 0.08 s, sys: 0.01 s, total: 0.09 s
Wall time: 0.09

or if I import it in the python interpreter, it's immediate.

The 0.3s is acceptable, 0.5s is quite a lot for me, and 1.4s is a lot. I am maybe too demanding, but it just annoys me to wait for 1.4s on the import time. I just get some idea to try, so I fire up Sage or SymPy and try it. I don't want to wait for 1.4s.

This can be fixed, one just needs to find modules that slow things down, and late import them, or just fix them in some way. And usually, one does it several times -- I think Sage did that twice, SymPy also I think did that twice already.

Saturday, December 22, 2007

People still (and will) use windows

I sometimes look at the download statistics of SymPy and the funny thing is that people download more the Windows installer, rather than the multiplatform source tarball. Part of this can be due to the fact, that SymPy is in Debian, Ubuntu and Sage, but still.

So here is how to make a windows installer in Debian:

$ ./setup.py bdist_wininst
[...]

and a file dist/sympy-0.5.9-hg.win32.exe is created. I only have regular python tools installed. So we just upload the exe to our site and that's it -- kind of scary, since I don't have means to test it.

But one time I tried that on one of brother's computers with windows and not only it worked, but also the 3D plotting using pyglet worked out of the box! If you are curious how it looks like:



So it has it's advantages to develop pure Python programs - they really run everywhere.

I think installing things like Python and playing with SymPy on windows must be pain (maybe people just download SymPy for windows and then run away with disgust). But just the fact that they try means, that people doing science do use windows a lot. And I don't think this is going to change any time soon. I also fully agree with Michael Abshoff's recent post about this issue.

Wednesday, December 5, 2007

Mérida - wrap up

From November 28 till December 2, 2007, I attended a Debian QA and release teams work session in Extremadura, which is an autonomous community of western Spain, that managed to install Debian on 90000 computers in every school (technically a Debian based distribution called gnuLinEx) and Junta de Extremadura also sponsored this meeting.

I took photos of all participants, see my first, second and third posts. When I arrived at the Madrid airport where we first met, I set myself a goal to remember all names and faces, so I used my blog to help me and I think I succeeded in the end. :)

See also our wiki page that we used prior and during the meeting.

So what did we do besides throwing candies?


Lucas Nussbaum will send a summary email soon about the meeting, so I'll just speak for myself:


I worked with Gonéri on svnbuildstat, that is a service for building packages and show statistics about lintian/linda/piuparts checks. It for example contains all packages of the Debian Python Modules Team (that I am a member of) and many others. We discussed and started to work on how to create robust buildbots, that can be installed as a regular Debian package with zero (if possible) configuration, so that many people can just install them without pain, thus providing a huge scalability to the project.

We wrote a preliminary patch to pbuilder for killing the build if it exceeds given memory/disk usage. I had to learn the internals of pbuilder and I lost quite some time squashing some stupid bug I caused while writing the patch.

I spent most of my time with svnbuildstat, mostly learning and discussing things. This will be important for the future, but to also have some real results, I also fixed some packages I comaintain:

Together with Kumar Appaiah we fixed the python-numpy package and I had it uploaded, then I learned how to work with quilt instead of dpatch to handle patches in Debian packages, thanks to Holger's webpage, that contains a nice tutorial. Then I switched from dpatch to quilt in python-scipy and backported a patch from upstream svn to fix a segfault bug and had the package uploaded.

Then I finished the Cython package and had it uploaded. Cython is a marvelous package to speed up Python programs and interface C/C++ programs. I greatly recommend to try. If you don't like it, you can try some of at least 10 other ways to wrap C code in Python. I also used quilt in there to backport a patch from the upstream Mercurial repository to implement parsing @classmethods. Quilt is really a pleasure to work with.

Impressions from the meeting

I've been using Debian since 2001 as my only operating system on all of my computers, so I am not a complete beginner. But it never occured to me I could get involved in Debian more than a user and an occasional bug reporter. What a mistake.

I started packaging new things and fixing packages that I need for my work and that didn't work. This got me involved quite a bit in Debian. But in Mérida it was the first time I could dring a beer (well, especially wine) with Debian Developers and I found out they are really cool people. They are all very skilled. Also something, that I love about Debian, is that the people involved in it share two common features, that are very important for them - respect to democracy and personal freedom. When I think about it, those are probably the first two items on my presonal list of values.

Of course, everytime there is a group of 1000+ people, there are good and bad people, more and less skilled, but important is the overall atmosphere - and that is as I described. I think Debian is truly unique. There is Gentoo, that has maybe 40 (?) active developers. There is Ubuntu, that has maybe 100 (?) developers, but it's basically a comercial distribution and there is not so many interesting work for non employees of Canonical. There is opensuse and fedora, where I am not sure about the numbers. The atmosphere in Debian can change in the future, one never knows, but as of the end of 2007, I think it's very cool to get involved.

Maybe it's not for everyone, but it's the right place for me.

Tuesday, December 4, 2007

Mérida - remaining photo

This post is dedicated to dato (blog), whose picture I forgot to take (thanks Cyril Brulebois for taking this one).

From left to right: Gonéri Le Bouder, me, Lars Wirzenius, Holger Levsen and Adeodato "dato" Simó!

Saturday, December 1, 2007

Merida

We spent the whole Friday hacking, Lucas will send a summary email soon. More people joined.

Ana Guerrero:


Amaya Rodrigo:


Mark Purcell:


Kilian Krause and César Gómez Martín


and Holger Levsen with red hair:


On Saturday we were again working and in the late afternoon we took a walk in Mérida, visited the famous ancient Roman monuments. Now it's 3:30am and we will soon go to Madrid and back home.

Thursday, November 29, 2007

Debian meeting in Merida, Spain

Right now, some Debian Developers (and also not yet Developers, like me:), are on
the work sessions in Extremadura, I am on the QA and release teams meeting.

We started in the morning with presentations (see also the schedule). Any comments and suggestions welcomed, please add comments below the post.

Lucas Nussbaum presenting:


Most of us:



And in details, names from left to right. Cyril Brulebois, Gonéri Le Bouder:


Luk Claes, Marc 'HE' Brockschmidt, Jörg Jaspert, Lars Wirzenius:


Fabio Tranchitella, Bernd Zeimetz, Mario Iseli, Luk Claes:


Filippo Giunchedi, Stefano Zacchiroli, Tzafrir Cohen, Simon Richter, Faidon Liambotis:


And again, so that Faidon is visible:

Wednesday, November 28, 2007

How to connect to the internet using T-Mobile and bluetooth

I have a laptop with Debian, cell phone (Nokia N70) with bluetooth and this howto describes how to connect to the internet.

Connect USB bluetooth dongle to the laptop and check it:

$ hciconfig
hci0: Type: USB
BD Address: 00:02:72:D2:23:12 ACL MTU: 310:10 SCO MTU: 64:8
UP RUNNING PSCAN ISCAN
RX bytes:686 acl:0 sco:0 events:22 errors:0
TX bytes:337 acl:0 sco:0 commands:21 errors:0

Scan for the phone:

$ hcitool scan
Scanning ...
00:19:79:86:EB:BC Nokia N70

Create a serial device for communicating with the modem in the phone:

$ rfcomm connect /dev/rfcomm0 00:19:79:86:EB:BC 3
Connected /dev/rfcomm0 to 00:19:79:86:EB:BC on channel 3
Press CTRL-C for hangup

(A dialog will popup on the phone asking me to allow the laptop to connect.)

Connect using wvdial:

$ wvdial
--> WvDial: Internet dialer version 1.56
--> Cannot get information for serial port.
--> Initializing modem.
--> Sending: ATZ
ATZ
OK
--> Sending: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
OK
--> Modem initialized.
--> Sending: ATDT*99***1#
--> Waiting for carrier.
ATDT*99***1#
CONNECT
~[7f]}#@!}!} } }2}#}$@#}!}$}%\}"}&} }*} } g}%~
--> Carrier detected. Starting PPP immediately.
--> Starting pppd at Wed Nov 28 12:58:55 2007
--> Pid of pppd: 4378
--> Using interface ppp0
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> pppd: �[08][06][08]�[10][06][08]
--> local IP address 172.24.171.97
--> pppd: �[08][06][08]�[10][06][08]
--> remote IP address 10.6.6.6
--> pppd: �[08][06][08]�[10][06][08]
--> primary DNS address 62.141.0.2
--> pppd: �[08][06][08]�[10][06][08]
--> secondary DNS address 213.162.65.1
--> pppd: �[08][06][08]�[10][06][08]

And that's it, I am connected.

Prerequisities

$ wajig install wvdial bluez-utils

And setup the /etc/wvdial.conf file:

$ cat /etc/wvdial.conf
[Dialer Defaults]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Modem Type = USB Modem
Baud = 460800
New PPPD = yes
Modem = /dev/rfcomm0
ISDN = 0
Phone = *99***1#
Password = doesnt_matter
Username = doesnt_matter
Stupid Mode = 1

Saturday, November 17, 2007

snapshot.debian.net rocks

Today during upgrade I got the gnome packages broken:

$ wajig install nautilus
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.

Since you only requested a single operation it is extremely likely that
the package is simply not installable and a bug report against
that package should be filed.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
nautilus: Depends: libexempi2 but it is not installable
E: Broken packages
$

I tried to download the source package of libexempi2, no luck (doesn't exist). Then I remembered I've heard about snapshot.debian.net, so I tried that and added:

deb http://snapshot.debian.net/archive pool exempi

to my /etc/apt/sources.list and voilà - it works!

$ wajig install nautilus
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libstdc++5 libpixman-1-0 libglut3 libsqlite0 lib64gfortran2 libqt4-sql
libntfs-3g4 libkdcraw1
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
fam libexempi2 libtrackerclient0 nautilus-cd-burner
Suggested packages:
tracker
The following NEW packages will be installed:
fam libexempi2 libtrackerclient0 nautilus nautilus-cd-burner
0 upgraded, 5 newly installed, 0 to remove and 46 not upgraded.
Need to get 1583kB of archives.
After unpacking 5632kB of additional disk space will be used.
Do you want to continue [Y/n]?
WARNING: The following packages cannot be authenticated!
libexempi2
Install these packages without verification [y/N]? y
Get:1 http://ftp.cz.debian.org sid/main fam 2.7.0-13 [69.0kB]
Get:2 http://ftp.cz.debian.org sid/main libtrackerclient0 0.6.3-3 [41.3kB]
Get:3 http://ftp.cz.debian.org sid/main nautilus 2.20.0-1 [639kB]
Get:4 http://snapshot.debian.net pool/exempi libexempi2 1.99.4-1 [290kB]
Get:5 http://ftp.cz.debian.org sid/main nautilus-cd-burner 2.20.0-1 [544kB]
Fetched 1583kB in 7s (199kB/s)
Selecting previously deselected package fam.
(Reading database ... 210515 files and directories currently installed.)
Unpacking fam (from .../archives/fam_2.7.0-13_i386.deb) ...
Selecting previously deselected package libexempi2.
Unpacking libexempi2 (from .../libexempi2_1.99.4-1_i386.deb) ...
Selecting previously deselected package libtrackerclient0.
Unpacking libtrackerclient0 (from .../libtrackerclient0_0.6.3-3_i386.deb) ...
Selecting previously deselected package nautilus.
Unpacking nautilus (from .../nautilus_2.20.0-1_i386.deb) ...
dpkg: warning - unable to delete old directory `/usr/share/mime/application': Directory not empty
Selecting previously deselected package nautilus-cd-burner.
Unpacking nautilus-cd-burner (from .../nautilus-cd-burner_2.20.0-1_i386.deb) ...
Setting up fam (2.7.0-13) ...
Starting file alteration monitor: FAM.
Setting up libexempi2 (1.99.4-1) ...
Setting up libtrackerclient0 (0.6.3-3) ...
Setting up nautilus (2.20.0-1) ...
Setting up nautilus-cd-burner (2.20.0-1) ...

Wednesday, November 14, 2007

SAGE Days 6

From November 9 till November 15 I attended SAGE Days 6 in Bristol, UK. It was a conference and a coding sprint for the SAGE project, which wants to build something comparable to Mathematica, Maple, Matlab and Magma built only from open source components without reinventing the wheel. Sage also provides excellent novel implementations for many mathematical algorithms (not found in Mathematica/Maple etc.).

I was invited to give a talk about SymPy, which is a Python library for symbolic mathematics, that we are doing.

Here are my notes from each day:

Saturday: one, two
Sunday: one, two
Monday, Tuesday, Wednesday: one

Impressions

Very positive. I started SymPy two years ago, because I wanted to play with symbolic mathematics in Python, see for example my presentation I gave at this conference for details. SAGE at that time was just able to do some mathematics things, but it was very weak in calculus, which is the SymPy's main domain. This has changed last half a year, when SAGE people managed to wrap Maxima in Python (which I thought to be completely impossible), so I started to follow SAGE development more closely. After the SD6, I must say I became very excited about the project.

One problem with mailinglists, IRC and other online interaction is that it's very difficult for me to get an impression about the people and the project. Being able to meet the developers and discuss with them face to face gives me the impression very quickly and very accurately.

By far the biggest guarantee, why it is worthy for me to contribute to SAGE, is the project leader, William Stein. He is very rational and pragmatic (I like these two properties) and after many discussions with him, I came to realize that he has basically identical views on the important things as I do and were I on his place, I would do the same decisions as he did and does. That's very nice, because I can concentrate my energy on things that I like to improve and don't have to worry about other things, because I know he will do it right.

The other SAGE developers are experts and with similar attitude as William has. It's enthusiastic to be among people who make things happen. For example one of the authors of Cython, Robert, implemented during SD6 a very nice HTML output, that shows Cython code, with colors according to how many Python API calls are called on that particular line, and by clicking that line it shows the corresponding C code.

SAGE project has a high aim and it stricly goes for it, without looking too much to the right or left, and that's how it should be. And it does produce a lot of very useful and high quality stuff along the way, for example Cython (probably the best wrapper for C/C++ things now, only pypy could possibly beat it, but that's still more a research project) or the SAGE notebook, which looks like a Mathematica notebook, but better and in a browser (together with a revision history, sharing, SSL encryption, etc.).

The only little problem is that currently SAGE developers are all mathematicians and as is well-known, mathematicians looks at mathematic from a very different prospective than physicists. :) And so I need calculus, advanced calculus and only when this is working, and working well, I can build on it some more advanced features. SAGE currently goes a little the other way - it has a lot of advanced features, from number theory, modular forms, elliptic curves, etc., but the basic calculus still needs a lot of improvements. SAGE wraps Maxima, because Maxima is quite fast, very well tested, so it works well. It's difficult to extend and written in LISP and that's very bad. That's where SymPy could help - it's in Python, very easy to extend, but currently slower than Maxima (rewriting parts of SymPy using Cython, or even C directly, will make it faster, hopefully as fast as Maxima or faster).

It's not yet in Debian, but SAGE people are working on it. It's not an easy task unfortunately.

Conclusion

SAGE is a very promising young project and I think it will succeed to provide an open source alternative to Maple, Matlab, Mathematica and Magma.

Monday, Tuesday, Wednesday

Monday

On Monday we, as usual, had a breakfast in the Marriott Hotel:



and went to the Heilbronn Institute:



that is right across the highest point in Bristol:



We had a presentation by Gregory Bard:



Then we had a discussion what things should be done in the following coding sprints. Then we coded and in the afternoon I had a presentation "SymPy: A Python library for symbolic mathematics".

Tuesday

In the morning Michael Abshoff gave a presentation about valgrinding a Python/Cython/C/C++ library application like Sage (which adds complexity and its own set of problems because of python). We spent the whole day coding, in the afternoon we did a quick session of lightning talks and in the evening we went to a pub, then we again coded till 1am, some even later.

Wednesday

In the morning we did a little coding and then had a wrap-up session:



Then William had a presentation "SAGE for number theorists" at the Heilbronn Seminar, after which most of the people went to pub, but I with Jaap Spies went to the Old City, as tourists. Then I visited a bookstore, read some books and then we had a last coding evening:


Sunday, November 11, 2007

Sunday afternoon

After a lunch we had a presentation by Clement Pernet about Fast Exact Linear Algebra, then David Loeffler talked about Computing Automorphic Forms for Unitary Groups using Sage:



and then Bill Hart gave a presentation about Algebraic Number Theory with Flint:



In the evening we went to an Indian restaurant and then we had a panel discussion
"The Future of Open Source Mathematical Software", moderated by John Cremona with panelists William Stein, Bill Allombert, Michael Abshoff, Dan Bernstein and me. Recording should be available after the conference.

In the evening, we went to the usual code sprint.

Sunday morning

After fighting with the famous British separated hot and cold water taps in the morning, I had the traditional bacon and eggs:



And it was delicious. Then we had the first presentation by Michael Brickenstein (see also the schedule):



After a break, we witnessed a marvelous presentation by Paul Zimmerman about MPFR, used by default by gcc and gfortran:



And we went to lunch.

Saturday, November 10, 2007

The rest of Saturday

We went to a common lunch at the institute, after which another 3 presentations followed. Robert Bradshaw on Cython, James Davenport on simplification in CAS and Dan Bernstein about elliptic curves. (Most people here are mathematicians, for example working on modular forms, cryptography etc.)

Then we had a conference dinner together, on which we discussed a lot of stuff - GPL2 vs GPL3 vs BSD pros and cons and problems and how to license SAGE and SymPy and whether "translating" something from C++ to Python is a derived work or not, I also asked William, the main author of SAGE sitting on my left hand side, how SAGE has started and about his future plans with it. Basically, we exchanged our experiences with working on an opensource project and how to motivate people to work on it, and how to do and especially not to do things. On my right hand side was Bill Allombert, one of the two authors of PARI/GP and also a Debian Developer, so we signed each other's gpg keys (that we use in Debian) earlier this afternoon. It's awesome, I am wearing my Debian t-shirt today, but I didn't expect to meet any Debian people here.

After a dinner we went to the math institute to have a coding sprint, some pictures using my cell phone from the sprint:











During the dinner and especially during the sprint I borrowed William's camera and recorded short interviews with conference participants. I am going to do more interviews tomorrow. I'll put the link to them later, when William puts them on the internet after the conference.

Downloading images from Nokia N70 over bluetooth in GNOME

is actually very easy. I connect the bluetooth dongle to the USB port on my laptop and:


$ wajig install gnome-bluetooth
$ gnome-obex-server
conn_request: bdaddr 00:19:79:86:EB:BC
conn_complete: status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '09112007196.jpg' Length 297132
** Message: Saving to '/home/ondra/Desktop/Downloads/09112007196.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC
conn_request: bdaddr 00:19:79:86:EB:BC
conn_complete: status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '10112007197.jpg' Length 289214
** Message: Saving to '/home/ondra/Desktop/Downloads/10112007197.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC
conn_request: bdaddr 00:19:79:86:EB:BC
conn_complete: status 0x00
** Message: Incoming connection from 00:19:79:86:EB:BC
** Message: Device 00:19:79:86:EB:BC is about to send an object.
** Message: File arrived from 00:19:79:86:EB:BC
** Message: Filename '10112007198.jpg' Length 287449
** Message: Saving to '/home/ondra/Desktop/Downloads/10112007198.jpg'
** Message: Incoming connection from 00:19:79:86:EB:BC


And then I just click on the image on my phone and say "send over bluetooth", it finds my computer, I say yes, and that's it. It ends up in my Downloads folder on the desktop.

Then I just upload the images to my blog, see an example how it looks like.

SAGE Days 6 - Friday and Saturday morning

Right now I am at the SAGE Days 6 conference.

On Friday, I took the plane



from Prague to Bristol, where we accomodated in the Marriott Hotel:



And went to have a dinner together in a Thai restaurant. On Staturday morning we went to our first session




to the math department:



where William:



and Martin:



gave an introductory talk about SAGE.

Monday, October 29, 2007

Mercurial vs git for managing Debian packages

I was used to managing all my packages in a Subversion repository, but now when we moved from Subversion to Mercurial in SymPy, I somehow prefer Mercurial for all my work now. :) We chose Mercurial, because I found it a little more polished than git, also the commands are the same as in Subversion.

I like the git-buildpackage, that it manages everything in one repository and that's it, and also that it's written in Python. One would expect that when Mercurial is written in Python, that hg-buildpackage would be too, but it's written in Haskell, so I cannot easily fix it. I filed a wishlist against the hg-buildpackage, whether it is a good idea to provide the same interface as git-buildpackage.

So for now I am going to try git for packaging things in Debian.