MTAPerformanceComparison: sendmailvs.postfixon*BSD BradKnowles SeniorConsultantforSnow,BV
[email protected] http://www.shub-internet.org/brad/papers/mtacomparison/
Entirecontentscopyright 2002byBradKnowles,allrightsreserved
Overview • Goal • Meta Information – Hardware Used – Software Tested – Tools Used – Methodology
• Test Results • Conclusions 2002-11-16
Copyright 2002byBradKnowles
2
Goal • Show you what it looks like to do MTA performance tuning – On as many *BSD platforms as I could
• Focus on the process, not the numbers or expected/desired outcome – You get to see (most) everything, warts and all 2002-11-16
Copyright 2002byBradKnowles
3
Goal • Basically, chapter 3 from the book sendmail Performance Tuning by Nick Christenson – See http://www.jetcafe.org/~n pc/book/sendmail/
2002-11-16
Copyright 2002byBradKnowles
4
MetaInformation • • • •
Hardware Used Software Tested Benchmark Tools Methodology
2002-11-16
Copyright 2002byBradKnowles
5
HardwareUsed • NetBSD – Twinhead “Twinstation 5G” (Sun SPARCstation 5 clone) – Not UltraSPARC, but the ancient SPARC 5
• • • •
OS: CPU: RAM: NIC:
NetBSD 1.6-RELEASE microsSPARC-II @ 110MHz 32MB real, 384MB virtual On-board “Lance” 10Base-T Ethernet & SBus QuadFastEthernet
• Unfortunately, disk drives died before testing could be performed (bug in NetBSD regarding SCSI tagged command queueing for old drives?) 2002-11-16
Copyright 2002byBradKnowles
6
HardwareUsed • FreeBSD – Compaq Armada 4131T • • • •
OS: CPU: RAM: NIC:
• HD:
FreeBSD 4.6.2-RELEASE Pentium 133 48MB real, 384MB virtual Asanté FriendlyNET AL1011 “Prism2” 802.11b WiFi PCMCIA 10GB IBM Travelstar 20GN
– 4200 RPM – 12ms avg. seek 2002-11-16
Copyright 2002byBradKnowles
7
HardwareUsed: FreeBSD
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
8
HardwareUsed • MacOS X (Part 1) – PowerBook G3 “Pismo” • • • • •
OS: CPU: RAM: NIC: HD:
MacOS X 10.2.1 PowerPC G3 @ 400MHz 1GB real, 2GB virtual Apple AirPort 802.11b WiFi 48GB IBM Travelstar 40GH
– 5400 RPM – 12ms avg. seek
2002-11-16
Copyright 2002byBradKnowles
9
HardwareUsed: MacOSX(Part1)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
10
HardwareUsed • MacOS X (Part 2) – PowerBook G4 (DVI) • • • • •
OS: CPU: RAM: NIC: HD:
MacOS X 10.2.2 PowerPC G4 @ 800MHz 1GB real, 2GB virtual Apple AirPort 802.11b WiFi 40GB Toshiba MK4018GAS
– 4200 RPM – 12ms avg. seek
2002-11-16
Copyright 2002byBradKnowles
11
HardwareUsed: MacOSX(Part2)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
12
SoftwareTested • Sendmail 8.12.6 – ftp://ftp.sendmail/org/pub/sendmail/sendmail.8.12.6.tar.gz
• Postfix 1.1.11 – http://www.postfix.org/ftp-sites.html
• Exim 4.10 – Tried every way I could think of, but could not get it configured to relay back to an IP address or port on the same machine 2002-11-16
Copyright 2002byBradKnowles
13
ToolsUsed • Disk Benchmarking – PostMark from Network Appliance • http://www.netapp.com/tech_library/3022.html
– Bonnie++ from Russell Coker • http://www.coker.com.au/bonnie++/
• SMTP Benchmarking – smtp-source/smtp-sink from Wietse Venema • Included with postfix distribution
– Postal from Russell Coker (with smtp-sink) • http://www.coker.com.au/postal/ 2002-11-16
Copyright 2002byBradKnowles
14
Methodology • Get disk subsystem baseline by performing benchmarking – For each platform & filesystem configuration • PostMark – See http://www.shub-internet.org/brad/FreeBSD/postmark.html
• Bonnie++ – Configured to test similar parameters » File sizes, numbers, & subdirectories according to the four PostMark tests 2002-11-16
Copyright 2002byBradKnowles
15
Methodology • For FreeBSD, we only test turning on or off “soft updates” – We do not test (turned on by default) • vfs.vmiodirenable • dirprefs • dirhash
– For data on all the possible combinations of the above, see “Recent Filesystem Optimisations in FreeBSD” at http://www.usenix.org/events/usenix02/tech/freenix/dowse.html
– For comparison data of Journaling vs. Soft Updates, see “Journaling Versus Soft Updates: Asynchronous Meta-data Protection in File Systems” at http://www.usenix.org/publications/library/proceedings/usenix2000/ general/seltzer.html 2002-11-16
Copyright 2002byBradKnowles
16
Methodology • For MacOS X – We test only • HFS+ – Apple’s Extended Hierarchical File System
• MFS – Memory-based filesystem created using RamBunctious 2.0
2002-11-16
Copyright 2002byBradKnowles
17
Methodology • For MacOS X – We do NOT test • UFS – IMO, Apple broke it while porting HFS+ to Mach – Many applications do not recognize it, in addition to MacOS 9 itself – “Classic” requires that boot device be HFS+ and MacOS 9 must be installed before MacOS X
• HFS+ with Journaling – Too new, introduced with MacOS X Server 10.2.2 – Initial reports indicate ~20% performance loss 2002-11-16
Copyright 2002byBradKnowles
18
Methodology • For MacOS X – Does not have a native “poll.h” or “libpoll” (required by postal) • Install manually from source http://www.clapper.org/software/poll/ • Install using “fink” – MacOS X package/port manager based on Debian “dselect”
– Unable to test Bonnie++ • Need further assistance from the author to find out why it is crashing
2002-11-16
Copyright 2002byBradKnowles
19
Methodology • Perform SMTP benchmarking – For each hardware & software configuration, test • Test direct source->sink connection – Demonstrate maximum possible performance
• Test source->relay->sink – Show relay performance
– Note • Testing only relaying, not local delivery • Test generated on, relayed through, and terminated on “loopback” network – Eliminate all question of network or NIC performance 2002-11-16
Copyright 2002byBradKnowles
20
TestResults • Chart data only – PostMark table data available from http://www.shub-internet.org/brad/FreeBSD/postmark.html
– Bonnie++ table data not yet available – smtp-source/smtp-sink data not yet available • `hostname` must be legal (I.e., does not end with “.”)
– Postal data not yet available 2002-11-16
Copyright 2002byBradKnowles
21
TestResults • Dumb “Driver” Scripts – My “runpostmark” script available at http://www.shub-internet.org/brad/runpostmark
– My “runbon” script available at http://www.shub-internet.org/brad/runbon
– My “runsmtp” script available at http://www.shub-internet.org/brad/runsmtp
2002-11-16
Copyright 2002byBradKnowles
22
TestResults: PostMarkSampleRun % cd /var/tmp % postmark PostMark v1.5 : 3/27/01 pm>set number 1000 pm>set transactions 50000 pm>show Current configuration is: The base number of files is 1000 Transactions: 50000 Files range between 500 bytes and 9.77 kilobytes in size Working directory: /var/tmp Block sizes are: read=512 bytes, write=512 bytes Biases are: read/append=5, create/delete=5 Using Unix buffered file I/O Random number generator seed is 42 Report format is verbose. pm>run
2002-11-16
Copyright 2002byBradKnowles
23
TestResults: PostMarkSampleRun Creating files...Done Performing transactions........Done Deleting files...Done Time: 497 seconds total 492 seconds of transactions (101 per second) Files: 26014 created (52 per second) Creation alone: 1000 files (500 per second) Mixed with transactions: 25014 files (50 per second) 24868 read (50 per second) 24880 appended (50 per second) 26014 deleted (52 per second) Deletion alone: 1028 files (342 per second) Mixed with transactions: 24986 files (50 per second) Data: 161.15 megabytes read (332.02 kilobytes per second) 168.38 megabytes written (346.92 kilobytes per second)
2002-11-16
Copyright 2002byBradKnowles
24
PostMarkTestResults PostmarkTest#1:1000Files&50kTransactions
90 8,000.00
2,500
80
7,000.00
70
2,000
6,000.00
60 1,500
50
Trans/Sec
KB/Sec
5,000.00
4,000.00
40
1,000
3,000.00
30
East West North
DataRead DataWritten Trans/Sec
2,000.00
20
500
1,000.00
10 PT P3 +U FS 2x P3 +S OF T 2x P3 +M FS U5 +U FS U5 +L og U5 +T M PF S PB G3 /H FS + PB G3 /M FS PB G4 /H FS + PB G4 /M Co FS mp aq +U Co FS mp aq +S of t
+D P3
inu
x
T OF
P3 /L
2nd Qtr 2x
+S
PP
ro
F6
30
30
S/
F3
NF
1st Qtr
NF
R5
S/
S/
R0
OD
S/
PF
2002-11-16
OD
TM
UF
0
S
0
S
0.00
Platform
3rd Qtr
4th Qtr
Copyright 2002byBradKnowles
25
PostMarkTestResults PostmarkTest#2:20kFiles&50kTransactions
90 3,000.00
700
80 70
600
60
500
50
400
2,500.00
1,500.00
40
300
30
200
1,000.00
20
500.00
Trans/Sec
KB/Sec
2,000.00
East West North
DataRead DataWritten Trans/Sec
100
10 Platform
/H
PB
G3
PB
U5
3rd Qtr
FS + G3 /M FS PB G4 /H FS + PB G4 /M FS Co mp aq +U FS Co mp aq +S of t
S PF
g
+T M
U5 +L o
FT 2x P3 +M FS U5 +U FS
+S O P3
2nd Qtr
2x
3+ U
FS
PT +D P3
2x P
x
T
inu /L P3
+S OF ro
F6 30
PP
30
S/
F3
NF
1st Qtr
NF
S/
S/
R5
R0 OD
S/
PF
2002-11-16
OD
TM
UF
0
S
0
S
0.00
4th Qtr
Copyright 2002byBradKnowles
26
PostMarkTestResults PostmarkTest#3:20kFiles&100kTransactions
90 2,500.00
80
700
70
600
60
500
KB/Sec
1,500.00
50
400
40
300
30
200
1,000.00
Trans/Sec
2,000.00
DataRead East DataWritten Trans/Sec West North
500.00
Platform
ft +S o
FS +U
mp
aq
aq
mp
+ FS PB
G4
/M FS
Copyright 2002byBradKnowles
Co
4th Qtr Co
FS
/H G4
+ FS
G3 /M
PB
PB
3rd Qtr
PB
FS MP
G3 /H
og +L
U5 +T
2nd Qtr
U5
63 0 ro +S OF T P3 /L inu x P3 +D PT 2x P3 +U FS 2x P3 +S OF T 2x P3 +M FS U5 +U FS
1st Qtr
PP
/F
33
S/
/F
NF S
NF S
/R
PF
2002-11-16
OD
OD S
TM
0
0
0 R5
10.000 0
100
S
20
27
PostMarkTestResults PostmarkTest#4:20kFiles,100kTransactions,&100subdirectories
90 6,000.00
1,400
80 70
1,200
60
1,000
50
800
5,000.00
3,000.00
40
600
30
400
2,000.00
20
1,000.00
Trans/Sec
KB/Sec
4,000.00
East West North
DataRead DataWritten Trans/Sec
200
10 ft mp aq +S o
FS
FS
4th Qtr
Copyright 2002byBradKnowles
Co
/M
Co mp aq +U
PB
3rd Qtr
PB G4
G4 /H
FS
+
FS G3
/M
FS +
Platform
PB
M +T
U5
2nd Qtr
PB G3 /H
S PF
og +L U5
FS U5 +U
FS 3+ M 2x P
P3 +S
2x P
2002-11-16
1st Qtr 2x
x inu
P3
/L
0
OF T
0
3+ UF S
0.00
28
PostMarkTestResults PostmarkTest#1:1000Files&50kTransactions
90 3,500.00
1,200
80
3,317.76 3,174.40
70
3,000.00
1,000
980
60
2,500.00
800
KB/Sec
40
600
1,638.40 1,566.72
1,500.00
485
30
492
Trans/Sec
50
2,000.00
East West North
DataRead DataWritten Trans/Sec
400
1,000.00
20 10
500.00
0
268.32 280.35 82
0.00
1st Qtr
PBG3/HFS+
2002-11-16
489.66 511.62 149
332.02 346.92
PBG3/MFS
2nd Qtr
200
235.40 245.96 72
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
Compaq+UFS
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
29
PostMarkTestResults PostmarkTest#2:20kFiles&50kTransactions
90 3,000.00
700
80
2,816.00 666
70
600
2,500.00
60
500
KB/Sec
50
400
40
1,500.00
1,495.04
349 1,392.64
300
30
Trans/Sec
2,000.00
East West North
DataRead DataWritten Trans/Sec
1,000.00
20
200
737.57
500.00
10 0
100 82.00
0.00
154.75 29
1st Qtr
PBG3/HFS+
2002-11-16
99.14
PBG3/MFS
187.09 36
2nd Qtr
38.95
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
73.51 15
Compaq+UFS
66.57
125.63 23
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
30
PostMarkTestResults PostmarkTest#3:20kFiles&100kTransactions
90 2,500.00
700
80
2,334.72 613
70
600
2,000.00
60
500
50
KB/Sec
1,500.00
400
40
1,177.60 310
300
1,000.00
30
Trans/Sec
1,628.16
East West North
DataRead DataWritten Trans/Sec
819.89
200
20 500.00
10 0
100 90.20
0.00
129.65 31
1st Qtr
PBG3/HFS+
2002-11-16
94.42
PBG3/MFS
135.72 33
2nd Qtr
42.68
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
69.72 100.21 22
61.35 15
Compaq+UFS
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
31
PostMarkTestResults PostmarkTest#4:20kFiles,100kTransactions,&100subdirectories
90 2,500.00
700
80 2,129.92
70
600
574
2,000.00
60
500
50
KB/Sec
400
40 300
1,015.93 277
1,000.00
30 708.74
20
Trans/Sec
1,484.80
1,500.00
East West North
DataRead DataWritten Trans/Sec
200
500.00
10 0
100
65.43
0.00
93.79 24
1st Qtr
PBG3/HFS+
2002-11-16
77.21
PBG3/MFS
110.68 28
2nd Qtr
38.81
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
55.63 14
Compaq+UFS
40.19
4th Qtr
Copyright 2002byBradKnowles
57.61 13
0
Compaq+Soft
32
TestResults: Bonnie++SampleRun % bonnie++ -d /var/tmp/bonnie -n 1:10k:0 -m compaq-soft -r 48m Writing with putc()...done Writing intelligently...done Rewriting…done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.02d ------Sequential Output------ --Sequential Input- --Random-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP compaq-soft 300M 2407 67 4584 39 2052 21 1948 53 3644 18 57.8 3 ------Sequential Create------ --------Random Create--------Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 1:10:0 402 31 +++++ +++ 832 49 564 46 +++++ +++ 1746 99 compaq-soft,300M,2407,67,4584,39,2052,21,1948,53,3644,18,57.8,3,1:10:0,402,31, +++++,+++,832,49,564,46,+++++,+++,1746,99
2002-11-16
Copyright 2002byBradKnowles
33
Bonnie++Results Bonnie++BlockI/O(SequentialCreate) 450
60% 402
400 50%
50% 47%
350
340
331 40%
250 31% 200
32%
31%
30%
181 23%
150
150 100
%CPUBusy
FilesCreated/Sec
300
FilesCreated/Sec %CPUBusy
20% 76
10%
50 0
0% compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
34
Bonnie++Results Bonnie++BlockI/O(SequentialRead) 800
50% 704 43%
700
67043%
69544%
45% 40%
600 500
30%
400
25% 20%
300
%CPUBusy
FilesRead/Sec
35%
FilesRead/Sec %CPUBusy
15% 200
165 10%
10%
100
5%
0
0% compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
35
Bonnie++Results Bonnie++BlockI/O(SequentialDelete) 2,500
70%
2,06260% 2,000
1,857 53% 50%
49%
48% 1,500
40%
30%
1,000 25%
%CPUBusy
52%
FilesDeleted/Sec
60%
FilesDeleted/Sec %CPUBusy
832 20%
500 215
10% 188
103
0
0% compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
36
Bonnie++Results Bonnie++BlockI/O(RandomCreate) 600 57% 500
60%
564
50%
48%
46% 40% 330 33%
350 35%
300
200
30% 24%
201
%CPUBusy
FilesCreated/Sec
400
FilesCreated/Sec %CPUBusy
20% 154
100
10%
78
0
0% compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
37
Bonnie++Results Bonnie++BlockI/O(RandomRead) 90
6% 81
80
79
80
5%
5%
5%
5%
70 4%
50
46 3%
40 30
2%
%CPUBusy
FilesRead/Sec
60
FilesRead/Sec %CPUBusy
2%
20 1% 10 0
0% compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
38
Bonnie++Results Bonnie++BlockI/O(RandomDelete) 2,000
120%
1,890
1,800
1,811
1,746
100%
99%
1,600
80%
1,200 1,000
60% 53%
53%
800 42%
%CPUBusy
FilesDeleted/Sec
1,400
FilesDeleted/Sec %CPUBusy
40%
600 400
20% 170
200
40 7%
397%
0
0% compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
39
TestResults: SmtpStoneSampleRun % sudo smtp-sink 127.0.0.1:25 1024 & % foreach I (1 2 3 4 5 6 7 8 9 10) foreach? /usr/bin/time smtp-source -l 10240 -f
[email protected] \ -t
[email protected] -m 1000 -s 512 127.0.0.1:25 foreach? end 8.06 real 0.22 user 1.65 sys 9.03 real 0.26 user 1.45 sys smtp-source: fatal: connect: Operation timed out 32.00 real 0.17 user 1.68 sys 13.08 real 0.19 user 1.43 sys 10.98 real 0.31 user 1.28 sys 9.02 real 0.24 user 1.52 sys 8.04 real 0.25 user 1.57 sys smtp-source: fatal: lost connection while reading server greeting 16.51 real 0.19 user 1.61 sys 10.99 real 0.19 user 1.53 sys 16.52 real 0.25 user 1.50 sys
2002-11-16
Copyright 2002byBradKnowles
40
SmtpStoneResults PowerBookG4SmtpStonePerformance(1000messages) 250.00
9.00 8.05
8.00
208.29
7.58
7.00
154.16
6.00
150.78
150.00
5.00 4.00 100.00
84.20
79.10
2.00
39.98 0.98 0.42
256
128
64
32
9.980.46
0.15
0.00 512
1.00
19.98 0.64
16
Msgs/Sec. Std.Dev.
3.00
68.19 50.00
StandardDeviation
MessagesperSecond
200.00
0.10 8
0.07 4
2
5.00 0.00 0.00 1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
41
SmtpStoneResults PowerBookG4SmtpStonePerformance(20000messages) 250.00
45.00
237.28
40.00 38.23 200.00
35.78
35.00
32.42
29.34
30.00
155.99
150.00 25.00
123.55 100.00
97.38
93.87
100.96
101.69
21.14 93.57 96.08
20.00
19.76 79.20
9.34
50.00
39.83
2048
1024
512
256
128
64
32
2.56
1.24
0.00 4096
10.00 5.00
2.58 8192
Msgs/Sec. Std.Dev.
15.00
5.10
16384
StandardDeviation
MessagesperSecond
32.92
188.79
16
0.00 8
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
42
SmtpStoneResults CompaqSmtpStonePerformance(1000Messages) 30.00
4.50
28.39 4.23
4.00 25.00
20.00
3.00 16.62
2.50
15.00 2.00 10.00
9.00
StandardDeviation
MessagesperSecond
3.50
Msgs/Sec Std.Dev.
1.50 1.00
5.00 0.50 0.17
0.02
0.00 4
2
0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
43
TestResults: PostalSampleRun % sudo smtp-sink 127.0.0.1:25 1024 & % postal -m 10k -p 64 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections 23:05,7651,41262,0,7715,0 23:06,7795,41656,0,7795,0 23:07,7847,42305,0,7847,0 23:08,7616,41528,0,7616,0 23:09,7670,41659,0,7670,0 23:10,7413,39922,0,7413,0 23:11,7765,41886,0,7765,0 23:12,7752,41960,0,7752,0 23:13,7423,39835,0,7423,0 23:14,7475,39697,0,7475,0 23:15,7458,39981,0,7458,0 23:16,7462,40332,0,7462,0 23:17,7281,39294,0,7280,0 23:18,7408,40022,0,7409,0 23:19,7595,40882,0,7586,0 23:20,7737,41991,0,7746,0 23:21,7646,41122,0,7646,0 ^C
2002-11-16
Copyright 2002byBradKnowles
44
TestResults: PostalSampleRun % sudo smtp-sink 127.0.0.1:25 1024 & % postal -m 10k -p 128 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections 23:27,8283,43988,0,8411,0 23:28,8965,48165,0,8965,0 23:29,8952,48100,0,8952,0 23:30,8785,46803,0,8785,0 23:31,7948,42811,0,7933,0 Server timed out on read. Server timed out on read. [ .. .. Deletia .. .. ] Server timed out on read. Server timed out on read. 23:32,8016,43367,28,8059,0 23:33,8960,47748,0,8960,0 23:34,9054,48507,0,9022,0 23:35,8737,47175,0,8769,0 23:36,9057,48586,0,9057,0 23:37,8800,46722,0,8800,0 23:38,8933,48092,0,8933,0 23:39,8969,48155,0,8955,0 23:40,8984,48576,0,8998,0 ^C
2002-11-16
Copyright 2002byBradKnowles
45
TestResults: PostalSampleRun % sudo smtp-sink 127.0.0.1:25 1024 & % postal -m 10k -p 256 -c 1 127.0.0.1 postal.user Usage: postal [-m maximum-message-size] [-p processes] [-l local-address] [-c messages-per-connection] [-r messages-per-minute] [-a] [-b [no]netscape] [-[z|Z] debug-file] [-s ssl-percentage] smtp-server user-list-filename conversion-filename Postal Version: 0.61 % postal -m 10k -p 32 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections Can't connect to 127.0.0.1 port 25. Can't connect to 127.0.0.1 port 25. Can't connect to 127.0.0.1 port 25. Can't connect to 127.0.0.1 port 25. Can't connect to 127.0.0.1 port 25.
2002-11-16
Copyright 2002byBradKnowles
46
PostalResults PowerBookG4PostalPerformance 160.00
400.00 378.02 145.77
140.00
350.00
120.00
300.00
100.00
250.00 79.69
80.00
200.00
164.94
60.00
150.00 39.92
40.00
9.98
27.51 10.06
0.00 128
64
32
16
Msgs/Sec Std.Dev.
100.00 19.97
20.00
StandardDeviation
MessagesperSecond
126.46
3.67 8
2.74
50.00 0.00
4
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
47
NotesforTesting • When testing real MTAs, you need to watch more than just the output from the testing program – Not unusual for MTAs to accept mail faster than they can process it, especially under heavy load
• Also watch – Mail queue size • mailq | grep '[0-9][0-9]:[0-9][0-9]' | wc -l
– CPU load • iostat 5
– Memory utilization • vmstat 5 2002-11-16
Copyright 2002byBradKnowles
48
NotesforPostfix • Make sure that you modify /etc/postfix/main.cf to be suitable – Need to boost • local_destination_concurrency_limit • default_destination_concurrency_limit
• Check /etc/postfix/master.cf to make sure there are no artificial limits on processes like smtp, smtpd, bounce, cleanup, rewrite etc… 2002-11-16
Copyright 2002byBradKnowles
49
NotesforMacOSX • Unfortunately, Apple broke iostat – Still mostly okay for what we want to watch (CPU user %, system %, and idle %)
• Apple doesn’t even provide vmstat – vm_stat is not the same program, although it is similar
2002-11-16
Copyright 2002byBradKnowles
50
PostfixSmtpStoneResults PowerBookG4PostfixSmtpStonePerformance(1000messages) 70.00
0.90 0.85
0.84 60.00
0.80 57.22 0.70
52.26 0.64
0.60 39.50
40.00
0.50 0.40
30.00 19.98
20.00
0.29
StandardDeviation
MessagesperSecond
50.00
Msgs/Sec Std.Dev.
0.30 0.20
10.00
9.92
0.13
4.99
0.08 0.00
0.10 0.00
32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
51
PostfixSmtpStoneResults Compaq+UFSSmtpStonePerformance(1000messages) 12.00
1.40 10.95 10.36
1.20
10.00
1.14
6.41
0.80
6.00 0.60 0.48
4.00
StandardDeviation
MessagesperSecond
1.00 8.00
Msgs/Sec Std.Dev.
0.40 0.33 2.00
0.20
0.00
0.00 4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
52
PostfixSmtpStoneResults Compaq+SoftSmtpStonePerformance(1000messages) 12.00
1.80 10.65 9.40
1.40
8.00
1.20
0.95
6.00
6.14 0.94
1.00 0.80
4.00
StandardDeviation
MessagesperSecond
1.60
1.55
10.00
Msgs/Sec Std.Dev.
0.60 0.40
2.00 0.20 0.00
0.00 4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
53
PostfixSmtpStoneResults CompaqPostfixSmtpStoneResults(20000messages/4Threads) 12.00
350.00 11.08
10.00
303.20
300.00
7.53 200.00
6.00 150.00
StandardDeviation
MessagesperSecond
250.00 8.00
Msgs/Sec Std.Dev.
4.00 100.00 2.00
50.00 17.72
0.00
0.00 compaq-ufs
compaq+soft
Platform-Filesystem
2002-11-16
Copyright 2002byBradKnowles
54
SendmailSmtpStoneResults PowerBookG4SendmailSmtpStoneResults(1000messages) 20.00
20.00 18.59 18.19
MessagesperSecond
16.00
18.00 14.89
16.00 14.36
14.00
14.00
12.00
12.00 9.69
10.00
10.00
8.00 6.00
8.00 5.37
4.97 4.75
4.00
1.92
16
8
6.00
2.00
1.47
0.68
0.37 32
Msgs/Sec Std.Dev.
4.00
2.00 0.00
StandardDeviation
18.00
4
2
0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
55
SendmailSmtpStoneResults PowerBookG4Sendmail+AsyncSmtpStoneResults(1000 messages) 20.00
1.40 18.01
18.00
1.20
1.16
16.00
MessagesperSecond
12.00
0.80 9.92
10.00
0.60
8.00 6.00
5.36
0.42
4.97
0.38
StandardDeviation
1.00
14.00
Msgs/Sec Std.Dev.
0.40
4.00 2.48
0.20
2.00
0.12
0.20 1.25
0.10
0.00
0.00 32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
56
SendmailSmtpStoneResults CompaqSendmail-UFSSmtpStoneResults(1000messages) 6.00
35.00 5.0231.04
5.00
30.00
4.00
25.00
3.92 3.65
20.00 3.00 2.49 14.08
15.00
StandardDeviation
MessagesperSecond
4.56
Msgs/Sec. Std.Dev.
2.00 10.00
9.67 1.00 4.17
5.00
4.04
0.00
0.00 16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
57
SendmailSmtpStoneResults CompaqSendmail+SoftSmtpStoneResults(1000messages)
6.00
90.00
6.61 82.20 5.91
80.00 72.11
70.00
MessagesperSecond
5.00
62.62
60.00
4.10 4.00
50.00
3.00
40.00
2.81 2.35
StandardDeviation
7.00
Msgs/Sec. Std.Dev.
30.00
2.00 20.37 1.00
10.93
20.00 10.00
0.00
0.00 16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
58
PerformanceTuning:Syslog • Problem – Local syslog can cause excessive load on busy mail system
• Solutions – Configure syslog to write to logs asynchronously • Perhaps via a “–” (dash) before the filename
– Run syslog on a central log server, across the network • Also allows you to perform more in-depth log processing, without interfering with mail services • Generally logs are much easier to manage centrally • Can also combine with other central syslog processing applications – E.g., Addamark Log Management System
2002-11-16
Copyright 2002byBradKnowles
59
PerformanceTuning:Syslog • Problem – Classic syslog uses UDP, can lose 75% or more of packets sent across the network
• Solution – Replace classic syslog with nsyslog, ssyslog, or syslog-ng and use TCP instead of UDP • Depending on your particular requirements for hardware, OS, environment, etc…. 2002-11-16
Copyright 2002byBradKnowles
60
PostfixSmtpStoneResults PowerBookG4(nosyslog)PostfixSmtpStone(1000messages) 70.00
3.00 2.80
60.89 60.00
2.50 52.59
MessagesperSecond
50.00
2.06
40.00
1.67
2.00
36.32 1.50
1.41 30.00
26.78
1.23
Msgs/Sec Std.Dev.
1.00
19.30
20.00
StandardDeviation
52.04
0.80 9.96
10.00
0.35
0.47
0.50
4.98
0.00
0.00 128
64
32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
61
SendmailSmtpStoneResults PowerBookG4Sendmail+Async+NoSyslogSmtpStoneResults (1000messages) 20.00
8.00
18.00
7.10
17.20
7.00
16.00 6.00 5.50 5.00
12.00 9.92
10.00
4.00
8.00 6.00
3.00 5.24
4.97 2.50
2.00
1.24 0.36
0.10
0.00 32
16
8
0.12 4
Msgs/Sec Std.Dev.
2.00
1.76
4.00
StandardDeviation
MessagesperSecond
14.00
2
1.00 0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
62
SendmailSmtpStoneResults CompaqSendmail-UFS-NoSyslogSmtpStoneResults (1000messages) 6.00
30.00
5.64
5.00
25.00
24.92
4.00
20.00
3.63
3.00
15.00 2.30 11.43
2.00
10.00
StandardDeviation
MessagesperSecond
4.46
Msgs/Sec. Std.Dev.
8.44 1.00
5.00 2.31
0.00
0.00 8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
63
Conclusions • This process can be difficult and timeconsuming to get right, but it pays off • Graphs & charts can help a great deal – But they don’t tell the whole story
• Software – Postfix is a lot faster than even I had thought – Wasn’t able to get sendmail tuned for the performance I’m confident it can deliver – Benchmarking tools (esp. Bonnie++ and postal) need more work to be useful 2002-11-16
Copyright 2002byBradKnowles
64
Conclusions • Use the Scientific Method – Try to set aside personal bias – Trust only what you can prove – Process • Generate Hypothesis • Generate test for Hypothesis • Test Hypothesis – Prove or Disprove
• If disproven, modify or generate new Hypothesis 2002-11-16
Copyright 2002byBradKnowles
65
RelatedWork • Nick Christenson – sendmail Performance Tuning book http://www.jetcafe.org/~npc/book/sendmail/
– “Performance Tuning Sendmail Systems” slides http://www.jetcafe.org/~npc/doc/performance_tuning.pdf
• Brad Knowles – “Sendmail Performance Tuning for Large Systems” slides http://www.shub-internet.org/brad/papers/sendmail-tuning/ 2002-11-16
Copyright 2002byBradKnowles
66
RelatedWork • Matthias Andree http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/vsqmail.html http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html
• Nakamura Motonori http://www.kyoto.wide.ad.jp/mta/eval1/eindex.html
• Ciprian Ascante http://www.benchmarks.dmz.ro/index.php?topic=LinMSB
2002-11-16
Copyright 2002byBradKnowles
67
