Sunday, December 19, 2010

Are you using the deadline scheduler? (Part 1)

There have been many posts about performance, benchmarking and the results. Many DBAs have talked in the past about the deadline scheduler, available in all modern Linux distributions.  The deadline scheduler is very effective for database systems, as it tries to prevent starvation of I/O requests.  To see what the net effect of using the deadline scheduler looks like, I am using a simple TPCC benchmark program, created by the Percona team, tpcc-mysql to measure the number of New-Order transactions per minute (TpmC).  All of the testing was performed using RedHat Enterprise Linux 5, using the standard ext3 file system and a combination of the default scheduler (cfq) and the deadline scheduler (dl):

CFQ:  845 TpmC
DL:  2145 TpmC

This is a huge difference in performance between the two schedulers. How do you know if you are using the deadline scheduler?  There are a few ways. The deadline scheduler can be enabled for all of your disks at boot time via the LILO or GRUB boot loaders. The scheduler can be changed by appending the string elevator=deadline to the boot line (file /boot/grub/grub.conf). If you issue the command cat /proc/cmdline this will display the kernel parameters that the system was booted with.

The second way of switching schedulers is to change it "on the fly." You can list all of the available schedulers, as well as determine which scheduler is current by using this command: cat /sys/block/drive/queue/scheduler where drive is the actual drive represented to your system (i.e. sda, hda).  The list of schedulers is displayed, and the current scheduler is noted within square brackets.  To change the scheduler, echo the name of the new scheduler to the meta-file from the previous command (as user root). Be careful when switching schedulers on the fly, as there is a possibility to hand the system.

In part 2, I'll go over my testing methodology and show how using the deadline scheduler with different types of RAID drive configurations and filesystem types affects the overall throughput of the system.

4 comments:

Andy said...

Some people also suggest noop scheduler. How does it compare to deadline? Have you benchmarked that?

Mark Robson said...

have you tested the noop scheduler? In my tests it usually won on throughput. Of course that's with a decent raid controller which most db servers will have. Don't take too much notice of io benchmarks on e.g. A spare laptop :-0

Partha Dutta said...

I have been performing my testing on a Dell 2950 with a Perc/5 RAID controller. I did perform a benchmark with the noop scheduler and there was very little difference in performance to the deadline scheduler.

Mark Robson said...

That seems consistent with what I observed then. I'm glad you did your tests on a raid controller. Some others have previously run performance comparisons on non-server hardware (sometimes laptops)