The future development of QTlab

[9x]

Hello fellow QTlab users!

This issue does not really concern the QTlab code, but rather the development process. I am working in Danneau group in Karlsruhe, Germany. We forked qtlab a long time ago and have been happily using it, adding some drivers and modifications to suit our needs. And while it seems that the main fork does not get any further updates since Reinier left Delft, the network graph looks like there are quite some other groups working the same way. But I have the impression that each of us is doing our own thing and there is little exchange or coordination. I feel that we as current users could spare some work if we would cooperate a bit more. And even more importantly, if we create different solutions for the same problems, the forks will drift apart, making it hard for new users to start using qtlab and eventually knocking out the project on long term.
I am not quite sure what I am asking for here, but maybe as a start, could the active users just "raise their hand" and comment this issue, so that we can get an overview on who is really working with qtlab and who would be interested in collaboration? I'm looking forward to hear your opinions on this issue!

With best regards,
Jens

[ggdeboo]

Hi Jens,
I am an active user based at UNSW in Australia. I'm maintaining a fork in which I mainly update the drivers/wrappers for the instruments that we use in our group. We use it on computers running Windows 7 or Scientific Linux.
I am interested in collaboration and developing new features, although I also don't have a clear view how QTlab should move forward.
Cheers,
Gabriele

[heeres]

Hi guys,

Great to hear that you're using QTlab! Hope it helps you out. I have indeed not been able to keep updating things here.

The biggest problem is that after leaving Delft and moving on to Yale I've started a rewrite of things to better suit my current experiments, so I'm using quite different code. In fact, we're rewriting that code again now and it is not as general as QTlab, but way more taylored to our experiments (and therefore more efficient).

Here are a few things I was unhappy with with QTlab:

• gnuplot plotting, although fast, is a bit cumbersome. While in Delft I started working on a replacement plotting client based on Chaco (big parts written by others in Delft). I didn't finish this, but I can send the work in progress to whoever is interested. (For my current experiments I just use matplotlib, as plotting speed is not a big issue).
• data format is not ideal. I think it would be good to move to hdf5 or something similar (of course with their own issues).

If there's anything I can do to support the current users I'd be happy to (as long as it's not too much work :-)

Cheers,
Reinier

[ggdeboo]

Another issue are the changes made to pyvisa after version 1.4. There have been several changes to the communication with instruments, which break the current implementation in QTlab.
Development on pyvisa has been relatively active, with the version in development being 1.8. Changes have been made to make pyvisa more pythonic and to give the user access to the ResourceManager from VISA. Since version 1.6 the legacy package vpp43 has been removed.

I'm interested in adapting the QTlab code to be compatible with recent pyvisa versions and I'm wondering how other users and developers are feeling about this, since it would break backwards compatibility. It would involve making changes to most instrument wrappers, many of whom I wouldn't be able to test myself.

[filipejc]

Hello,

Any news on Qtlab development?

BR,
Filipe Coelho

[wakass]

Hi people,

Great to see more people interested in qtlab and how it can improve. I thought I'd just toss my hat in the ring here and share what we've been working on what our experiences are. It would be great if the community could pool its resources and really take qtlab to the next level!

Our fork, neqstlab, mostly includes an extra library called parspace, a virtual parametric instument, gzipping of measurement files, and the necessary bugfixes.
We've been using it here at the UTwente in the Netherlands for about 3 years now to great satisfaction and have built quite a bit of software architecture around it. Part of that is our plotting library tessierplot, which takes qtlab generated measurement files en masse and allows an easy overview of all your data in an ipython/jupyter notebook.

Besides running on windows, we've recently managed to get qtlab up and running on linux running pyvisa 1.8 with the pyvisa-py backend (you can find modified code on the linux branch of neqstlab), which seems to run just fine in most cases. There were some issues with gnuplot crashing but that was fixed by gnuplot 5.1. This branch 'should' also run on windows but hasn't been tested for all instrument ofcourse.

I see the following opportunities for improvement (big and small):

User experience

• More expressive scripting/definition of measurement. Without further abstraction, measurement scripts can be an amalgam of for and while loops which does not work well for maintainability or first-time users. Our parspace library is a first attempt at dealing with this by setting the axes and grid of your parameter space, and being agnostic about how it is measured. This is potentially a useful tool when exploring a big parameter space. A consistent way to measure would also allow xml's of a specific measurement to be created. Which is great for data provenance.
• More metadata/More support for saving the exact way data was measured. Currently we're just saving the measurement script specifically because there are parameters which the .set file does not update(bug?) or capture, e.g. gains or conversions which are not controllable by electronic means.

Threading

• Measurement timing. Varying degrees of delay in the code at various places (caused by live plotting in our case) cause delays. In some cases it can lead to measurement artifacts (think capacitive charging).
  This goes hand in hand with the way sweeping is handled internally in qtlab, e.g. when the step taken by an instrument is < maxstep the sweep proceeds without any explicit timing at all.
  I think more granularity in timing control would be a great(TM) thing.
• GUI lockup/Threading/Feedback. Currently, when an instrument is swept over values bigger than maxstep, the gui locks up, and no feedback is given of the current value. In general, qtlab requires the issuing of qt.msleep in order for the GUI to process its queue.
  This, and from what I've seen in the source of sharedobject (subclassed from gtk's gobject), the GUI loop is very tightly integrated into measuring. I'd say the way forward here then would be disentangling the gtk message loop and the measurement loop. Perhaps Reinier could comment more on this specific issue.
• A more modern framework for handling GUI. The qt library would seem the appropriate choice, though there might be some namespace clashes ;)

Data format

• A better data container is indeed a good start. Perhaps a mix of XML and binary would lend itself well.
• Explicit support for units! This would allow great things to be done in analysis. As far as I can see the pint library is the standard in this area.

Long long term

• Better integration with an analysis framework. It would be great if the plotting/analysis part after data acquisition could be fed back exactly one to one. i.e. take a rough scan, zoom in to desired area, increase resolution, send that to remeasure. Perhaps qtlab could be better integrated into a jupyter notebook e.g.

Lots of opportunity I believe to make qtlab better still! Let me know what you guys think.

Cheers,

Chris

[ijvm]

Dear all,

This issue has been inactive for one year, so it is fair to say that any cooperative development of QTlab has not flourished. Nevertheless, I feel compelled to follow on the initiative of @9x and on the very enthusiastic comment of @wakass . I know there are still several labs actively using QTlab, certainly many more than the few ones which have public fork repositories.

I say this from personal experience. I started using QTlab a couple of years ago, and I helped to introduce its use in the transport/graphene groups at NUS (Singapore), and Groningen (the Netherlands). I also know that at the (former) lab of @wakass (Twente, Netherlands) they still use it (I was there long time ago and enquired today about it with former colleagues). Now, together with a former postdoc from UNSW (which is the lab of @ggdeboo ), I am introducing the use of QTlab in Manchester (UK).

For the time being I have only copied @heeres master and adapted/developed simple instrument wrappers, which we can do on our own. But I see here already some students interested in working and developing QTlab. So, with a view towards its long-term sustainability, I make again a call for active users (e.g. anybode from Delft?) to come forward and continue a discussion on sharing forks and collaborating.

Cheers,
Ivan

[AdriaanRol]

Hi @ijvm in Delft we have mostly moved over to QCoDeS. QCoDeS is an open source project that is supported by Microsoft and is heavily inspired by QTLab and as such shares much of the same concepts. I think it addresses some of the more fundamental flaws of QTLab, most notably the ease of installing and maintaining it.

I would personally recommend moving over to QCoDeS, especially when starting up a lab as that package is still quite active. However, as with any open source project it also has it's own particular flaws.

[ijvm]

@AdriaanRol thank you for bringing up QCoDeS, I was not aware of it. I see it is indeed similar in spirit to QTLab. A first step is to encourage students here to transition from the LabView 'point and click' culture towards learning how to handle (Python) scripts. For this purpose QTLab is enough.

I will certainly check QCoDeS later, and if it remains active most probably transition to it. In the meantime, could you please briefly state what are, in your opinion, the main differences between the two projects? (Besides their level of activity). Thanks.