illustrated how I managed to throttle the operations towards cassandra, based on asynchronous operations & batched operations.
there’re 3 primary operations, `save`, `delete` & `load`, all three use the same `primary key` as `path`, and all cassandra operations are modeled as akka messages, and sent to a single `Actor`:`ChannelManager` in our case. The manager is capable of grouping operations, and merge multiple operations into fewer, much fewer. And the manager executes merged operations at fixed intervals (10ms).
the downside is obviously that we’ll have some extra latency (10ms in worst case) for the operations, but given our stats that cassandra operations are mostly over 100ms or 200ms, this <10% increase doesn’t hurt much. on the other hand, QPS is now bounded by the number of intervals, 100 in this case, multiplying the groups it merges, [1, x], x is a bit undetermined due to possible grouping conflicts (path collisions), but should be a small integer after all. and therefore we get the throttle.
here’s some metrics collected by cassandra client in prod: