Tod, 

I've wondered for a little while if we should be working with a message queue service like Kafka or RabbitMQ for some backend operations related to joins. I'm more than a bit beyond my technical expertise here; me saying "joins are slow; I'll research MQ services to see if they would allow more data-duplication across services with eventual consistency" is like me saying, "driving to the grocery store is slow; I'll research harrier jet planes." I can't speak for other developers, but I wonder if they are in a similar situation, i.e. as your post comments they have a strong background in RDBMS-backed systems with full ACID compliance, but not in sharded datastores and microservices. We might know enough about these technologies to know how they're used, but not enough to feel confident about making recommendations about architecture, or to contribute to an RFC or whitepaper. 

To that end, targeting a few key developers who have more experience with distributed systems would be a good way to move that discussion forward, but the question is how we can identify those folks. Probably they would be members of the back-end team. Jakub, do you have any suggestions? I don't know that team as well. 

Zak



On Mon, Jul 23, 2018 at 7:53 AM Tod Olson <tod@uchicago.edu> wrote:
TC,

On Wednesday, after consulting with Jakub and Vince, I posted to Discuss about distributed updates and eventual consistency and mentioned it on the #development channel. The point is to get some conversation going about how to deal with these issues, and whether the project needs to have some more support for the relevant micro services techniques or come to some best practices. There were some positive reactions in Slack, though no conversation, neither in Slack nor Discuss. The topic fits well with Vince's RFC for an asynchronous event service.

Does it make sense to prod the dev community a little more to have a discussion on how we manage updates with distributed data store? And if so, what would be an effective way to do it? Target a few key developers?

Alternately, I could wait for Vince to have his RFC ready for review, but it seems like this sort of discussion could lay groundwork for more enthusiastic support for the asynchronous event service.

-Tod

To unsubscribe from this list please go to http://www.simplelists.com/confirm.php?u=ziEpDIpEAdOqeytbJdCltsFABsHplgyC