Plone 6 support?

[gforcada]

Is anyone still interested/using in this package? 🍀

I see that there is a https://github.com/collective/collective.taskqueue/tree/python3_compatibility started by @cekk and with plenty of work from @datakurre are you running it on python 3 with Plone 5.2/6.0 ? 🤔

Did you move to something else? what? 😄

[datakurre]

@cekk did fix syntax and imports. I continued to port this to work with my ZSever fork for Python 3, but that failed to get any momentum. I guess, nobody ever had real need for websockets in core Plone.

Our use cases for this were places where external task scheduler looked overkill. But with WSGI, external task scheduler is mandatory, so upgrading this would need to make choices and have opinions on external task schedulers that don't want to make. Also, c.taskqueue allowed "hidden browser views" that were only visible for taskqueue scheduler and not regular HTTP requests, and that is no longer an option with WSGI.

Beyond c.taskqueue, we've been solving our async and task scheduling needs on Plone 6 with custom code in Volto layer (because Volto is running on async server) or using Camunda 7 as external task / integration process orchestrator.

[gforcada]

Nice to hear/read from you! 👋🏾 😄

Regarding websockets, that might be yesterday news, because at the Alpine City Sprint, there were quite a few persons asking for that, at least @ericof was looking for that, we at @derFreitag are also interested in it as well

[datakurre]

Real use-cases for websockets would help.

I doubt ZServer could have any future now when that everyone has already migrated to WSGI.

Still, I wonder, if anyone is really using some other WSGI server than the default waitress. Which actually backported asyncore from Python 2. AFAIK, the same asyncore, which did run the original ZServer...

Simpler CMS frameworks already support ASGI, but I don't know it enough to say, if Zope could have ASGI future. It would not take away our issue with expensive ZODB connections requiring a lot of memory and warmup.

That all said, it is well possible the same possible solution could help both task queue and websocket use cases. With Volto, the obvious solution would be to build the solution onto Volto server. Classic would require some proxy solution. My ZServer fork's websocket support was based on Twisted, Autobahn and ZMQ, and I still don't know, how large amount of pub/sub subscribers it could really have scaled. (We already had RabbitMQ based solutions for hi-frequency event streaming [video encoding progress] and chats.)

https://datakurre.github.io/ploneconf2019/zserver-reloaded-with-http2-and-websocket-support.pdf

https://www.youtube.com/watch?v=tXeVdt6h80E

[datakurre]

@gforcada I heard, you are considering https://plone.org/news-and-events/events/sprints/plone-midsummer-sprint-2023

I can work on any one topic there. Just pick our fight :) But I assume Playwright has higher priority.

[davisagli]

The discussion of websockets might belong better somewhere else, since it's sort of a different topic from task queues. But anyway, some use cases:

• Show an alert message if there is an update to the page that is currently being viewed
• Show new comments without needing to reload (danger, probably a bad idea from a human-behavior-on-the-internet perspective)
• Broadcast an alert to everyone currently viewing the site

Simpler CMS frameworks already support ASGI, but I don't know it enough to say, if Zope could have ASGI future. It would not take away our issue with expensive ZODB connections requiring a lot of memory and warmup.

You can have an async loop in one thread plus a thread pool with a limited number of threads to handle tasks which use the ZODB. There are patterns from frameworks like FastAPI and Django for running a synchronous task in the thread pool initiated from the async code, or starting an async task from the synchronous code. But you would have to be careful about sending persistent objects across those boundaries because bad things would happen if the async code was trying to use another thread's _p_jar.

[datakurre]

@davisagli Thanks. These are related discussion architecturally. I built collective.taskqueue to have good enough asynchronous task queue for casual tasks directly in Plone without an additional processes. This fit well into ZServer, which was a single server built to run multiple concurrent services on asyncore. But not so well with WSGI Plone without breaking WSGI compliancy.

Is it still a feature that Plone Classic could run standalone? Should we give up on the idea of having task queues or websockets built in Plone, and focus on supporting external components instead? There has been similar discussion with image scaling. A few years ago, I did a proof-of-concept that we could run asynchronous multi-threaded image-scaling dispatched from WSGI requests, but that didn't stop discussion for proxying images with an external service like thumbor.

Show an alert message if there is an update to the page that is currently being viewed
Show new comments without needing to reload (danger, probably a bad idea from a human-behavior-on-the-internet perspective)
Broadcast an alert to everyone currently viewing the site

As these are all read-only-use-cases, we could be handled simply (:crossed_fingers:) by dispatching events from Plone to external websocket broker. Maybe even with a reusable content rule.

I wonder, if just having Redis could work for both task queues and websockets.

For task queues, standalone workers would still be needed to consume queues. I recall seeing David's simple examples, where tasks are executed in almost similar context to browser views. But why couldn't a worker like that could use ZPublisher to get identical context to normal requests. (collective.taskqueue mapped tasks to browser views only visible for requests tagged with a specific marker interface to keep them hidden from real HTTP requests; browser views were used to run tasks in "familiar" environment)

On websockets, authorization has some unsolved issues. In my demo 2009, I did rely on an option for each message being checked against a cacheable guard before being delivered. Unfortunately, that would not work with just Redis, but would require one more service (or be embedded in Volto server).

https://datakurre.github.io/ploneconf2019/zserver-reloaded-with-http2-and-websocket-support.pdf

[gforcada]

Sorry, now I'm catching up 😅

@davisagli examples are mostly our use cases, also for Web Push Notifications (I work for a newspaper at the end 🗞️ 🤣 ).

Regarding a Zope ASGI server or an out of context: IMHO, having an internal solution (batteries included) that can eventually be externalized would be the best (like ZCatalog but then if you need to scale use collective.solr or collective.elasticsearch or collective.taskqueue with internal queues or a redis backend).

As for the sprint, yes that's my idea, I need to book the flights 😅 but my idea is to be there, yes 🎉 and playwright might be a great idea 🌟

[djay]

Regarding a Zope ASGI server or an out of context: IMHO, having an internal solution (batteries included) that can eventually be externalized would be the best (like ZCatalog but then if you need to scale use collective.solr or collective.elasticsearch or collective.taskqueue with internal queues or a redis backend).

@gforcada c.taskqueue is only partly out of the box async tasks. Out of the box it only lets run async tasks on the same instance. if you want a seperate worker instance you need redis. p.a.async gave you a complete out fo the box solution with nothing external since it used zodb to queue between instances. But I agree having async task primitives in plone is long over due. Having same server tasks is perhaps good enough compromise but should be clear on what you are getting

@datakurre at least for our setup we have no problem going back to zpublisher, esp if it got us easier tasks, c.futures and websockets. I don't think wsgi gave us anything other than the theory we could be more "standad". Others might have made use of middleware?