108 points | 6d ago | Discuss on Hacker News | Back to Radar
I hadn't heard of it, and thought to share it here. I'm not sure why people are upvoting it - but it's not impossible that others are interested in this topic as well..! Cheers
At the end we moved from a cluster to a single instance running on a over provisioned machine, that worked pretty well.
So we had our application talking to two nodes of a cluster, both working normally without talking to each other (nevermind the fact that the connectivity between the machines was fine). Some workers picked up jobs, others didn't. IIRC the app read requests from a queue and wrote replies to another one, so we'd have half of the jobs being picked up, and then half of the replies not being picked up (ergo, 75% of the requests failed). It was bizarre and nothing seemed to indicate there were issues with RabbitMQ itself. I don't even remember how we figured it out at the time.
It's been nearly 10 years, I'm sure that it's better, that I was inexperienced, that the original setup wasn't great. But that left me so sore on it that I have actively avoid it ever since.
Bonus points for having dealt with this while Django Celery apparently had two independent configuration mechanisms, and both were being used in this app. Fun times.
brokerless is nice niche for when you have pretty constrained use case but just not traversing network well (zmq requires multipath for any sensible patterns) already roots many use cases out, as is the authentication
There are a bunch of options but the one I settled on is zigbee2mqtt, which requires a broker for message-exchange. Once it was setup it was pretty trivial to write simple clients to connect to the mosquitto instance and read/modify device state.
(e.g. I wanted something that would turn on my bedside light at 6am, and something else that would trigger some flashing lights when my laundry machine was finished and I did that via some external processes.)
I looked at MQTT options earlier last year, and the best one was Mochi ( https://github.com/mochi-mqtt/server ), written in Go.
I added TLS support with Let's Encrypt provisioning to it: https://github.com/Cyberax/lenc-mochi
For services that can't be avoided, I try to limit their exposure. I run Postgres with strictly controlled network access. Same for Redis.
What else? I'm also using Hickory for DNS. I replaced the stock OpenSSH with a version compiled using Fil-C.
i picked mosquitto but had to ditch it for AWS IoT Core and have not looked back.
HA is a more difficult thing to solve, I believe Cedalo sell a version with HA support which may solve all this but the open source version doesn’t really have any good answers. The usual approach seems to be setting up topic replication between all your nodes and putting a load balancer in front.
Thankfully the system I’m responsible for can tolerate outages so we just work on the basis that we have regular snapshots and can stand up a new instance if the one handling traffic fails, with the fault tolerance sitting on the client side with messages being queued for transmission when the connection comes back.
I replaced Mosquitto with Vernemq to get websocket support and other things. Vernmq also does clustering.
Comments are loaded live from Hacker News and are not stored by Mid or Real.