tame cassandra with akka

illustrated how I managed to throttle the operations towards cassandra, based on asynchronous operations & batched operations.

there’re 3 primary operations, `save`, `delete` & `load`, all three use the same `primary key` as `path`, and all cassandra operations are modeled as akka messages, and sent to a single `Actor`:`ChannelManager` in our case. The manager is capable of grouping operations, and merge multiple operations into fewer, much fewer. And the manager executes merged operations at fixed intervals (10ms).

the downside is obviously that we’ll have some extra latency (10ms in worst case) for the operations, but given our stats that cassandra operations are mostly over 100ms or 200ms, this <10% increase doesn’t hurt much. on the other hand, QPS is now bounded by the number of intervals, 100 in this case, multiplying the groups it merges, [1, x], x is a bit undetermined due to possible grouping conflicts (path collisions), but should be a small integer after all. and therefore we get the throttle.

here’s some metrics collected by cassandra client in prod:

before:

50thPercentile 42.852719
75thPercentile 90.72111274999999
95thPercentile 218.66267364999996
98thPercentile 268.9225570199998
999thPercentile 1943.6772506390055
99thPercentile 312.0279865900006
Count 2546151
DurationUnit milliseconds
FifteenMinuteRate 37.40239232827823
FiveMinuteRate 38.086199995828565
Max 1989.103602
Mean 70.51469555642024
MeanRate 27.84432567185669
Min 15.665704
OneMinuteRate 38.55207382895939
RateUnit events/second
StdDev 91.52011232439068

after:

50thPercentile 48.025407
75thPercentile 92.52239525
95thPercentile 230.81847554999993
98thPercentile 288.62097035999926
999thPercentile 1468.7712959410026
99thPercentile 351.78678019000023
Count 78777
DurationUnit milliseconds
FifteenMinuteRate 24.989111985499903
FiveMinuteRate 25.78200216101393
Max 1489.4018979999998
Mean 74.30991756128405
MeanRate 25.09424376174075
Min 16.072432
OneMinuteRate 25.768704045339028
RateUnit events/second
StdDev 87.9649453389184
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s