  Linux SMP FAQ
  David Mentr, David.Mentre@irisa.fr
  v0.22, 17 april 1998

  This FAQ review main issues (and I hope solutions) related to SMP con-
  figuration under Linux.
  ______________________________________________________________________

  Table of Contents


  1. Introduction

  2. Questions related to any architectures

     2.1 Kernel side
     2.2 User side

  3. Intel architecture specific questions

     3.1 Why it doesn't work on my machine?
     3.2 Possible causes of crash
     3.3 Motherboard specific information
        3.3.1 Motherboards with known problems
        3.3.2 Motherboards with no known problems

  4. Useful pointers

     4.1 Various
     4.2 SMP specific patches

  5. Glossary

  6. List of contributors



  ______________________________________________________________________

  11..  IInnttrroodduuccttiioonn

  Linux can work on SMP (Symetric Multi-Processors) machines. SMP
  support has started with the 2.0 family and has been improved in the
  2.1 (future 2.2) saga.


  FAQ maintained by David Mentr (David.Mentre@irisa.fr). The latest
  edition of this FAQ can be found at
  http://www.irisa.fr/prive/mentre/smp-faq/.


  If you want to contribute to this FAQ, I would prefer a diff against
  the SGML version http://www.irisa.fr/prive/mentre/smp-faq/smp-
  faq.sgml> of this document, but any remarks (in plain text) will be
  greatly appreciated.


  This FAQ is an improvement of a first draft made by CChhrriiss PPiirriihh.


  All information contained in this FAQ is provided "as is." All
  warranties, expressed, implied or statutory, concerning the accuracy
  of the information of the suitability for any particular use are
  hereby specifically disclaimed. While every effort has been taken to
  ensure the accuracy of the information contained in this FAQ, the
  authors assume(s) no responsibility for errors or omissions, or for
  damages resulting from the use of the information contained herein.


  22..  QQuueessttiioonnss rreellaatteedd ttoo aannyy aarrcchhiitteeccttuurreess


  22..11..  KKeerrnneell ssiiddee


  1. DDooeess LLiinnuuxx ssuuppppoorrtt mmuullttii--tthhrreeaaddiinngg??  IIff II ssttaarrtt ttwwoo oorr mmoorree
     pprroocceesssseess,, wwiillll tthheeyy bbee ddiissttrriibbuutteedd aammoonngg tthhee aavvaaiillaabbllee CCPPUUss??

     Yes.


  2. WWhhaatt kkiinndd ooff aarrcchhiitteeccttuurreess aarree ssuuppppoorrtteedd iinn SSMMPP??


     FFrroomm AAllaann CCooxx:
        SMP is supported in 2.0 on the hypersparc (SS20, etc.) systems
        and Intel 486, Pentium or higher machines which are Intel
        MP1.1/1.4 compliant.

        SMP support for UltraSparc, SparcServer, Alpha and PowerPC
        machines is in progress in 2.1.x.


     FFrroomm RRaallff BBcchhllee:
        MIPS, m68k and ARM does not support SMP; the latter two probly
        won't ever.

        That is, I'm going to hack on MIPS-SMP as soon as I get a SMP
        box ...


  3. HHooww ddoo II mmaakkee aa LLiinnuuxx SSMMPP kkeerrnneell??

     Uncomment the SMP=1 line in the main Makefile
     (/usr/src/linux/Makefile).

     AND

     enable "RTC support" (from RRoobbeerrtt GG.. BBrroowwnn). Note that inserting
     RTC support actually doesn't afaik prevent drift, but according to
     a discussion [Robert G. Brown] remember from a year ago or so it
     can prevent lockup when the clock is read at boot time.

     AND

     do NOT enable APM!  APM and SMP are not compatible, and your system
     will almost certainly (or at least probably ;)) crash under boot if
     APM is enabled (JJaakkoobb OOeesstteerrggaaaarrdd). AAllaann CCooxx confirms this : 2.1.x
     turns APM off for SMP boxes.  Basically APM is undefined in the
     presence of SMP systems, and anything could occur.



     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to make modules and make
     modules_install (from AAllaann CCooxx).


  4. HHooww ddoo II mmaakkee aa LLiinnuuxx nnoonn-SMP kernel?

     CCoommmmeenntt the SMP=1 line in the Makefile (and not set SMP to 0).

     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to make modules and make
     modules_install.



  5. HHooww ccaann II tteellll iiff iitt wwoorrkkeedd??


      cat /proc/cpuinfo




  Typical output (dual PentiumII):

  ______________________________________________________________________
  processor       : 0
  cpu             : 686
  model           : 3
  vendor_id       : GenuineIntel
  stepping        : 3
  fdiv_bug        : no
  hlt_bug         : no
  fpu             : yes
  fpu_exception   : yes
  cpuid           : yes
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
  bogomips        : 267.06

  processor       : 1
  cpu             : 686
  model           : 3
  vendor_id       : GenuineIntel
  stepping        : 3
  fdiv_bug        : no
  hlt_bug         : no
  fpu             : yes
  fpu_exception   : yes
  cpuid           : yes
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
  bogomips        : 267.06
  ______________________________________________________________________




  6. WWhhaatt iiss tthhee ssttaattuuss ooff ccoonnvveerrttiinngg tthhee kkeerrnneell ttoowwaarrdd ffiinneerr ggrraaiinneedd
     lloocckkiinngg aanndd mmuullttiitthhrreeaaddiinngg??

     2.1.x has signal handling, interrupts and some I/O stuff fine grain
     locked.  The rest is gradually migrating. All the scheduling is SMP
     safe


  7. DDooeess LLiinnuuxx SSMMPP ssuuppppoorrtt pprroocceessssoorr aaffffiinniittyy??

     No and Yes.  There is no way to force a process onto specific CPU's
     but the linux scheduler has a processor bias for each process,
     which tends to keep processes tied to a specific CPU.




  22..22..  UUsseerr ssiiddee


  1. DDoo II rreeaallllyy nneeeedd SSMMPP??

     If you have to ask, you probably don't. :)


  2. HHooww ddooeess oonnee ddiissppllaayy mmuuttiippllee ccppuu ppeerrffoorrmmaannccee??

     Thanks to SSaammuueell SS.. CChheessssmmaann, here is some useful utilities:

     CChhaarraacctteerr bbaasseedd::
        http://www.cs.inf.ethz.ch/~rauch/procps.html

        Basically, it's   procps v1.12.2   (top, ps, et. al.)  and some
        patches to support SMP.


     GGrraapphhiicc::
        xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included)
        have the /proc/stat/cpuX entry.

        The official homepage for xosview is:
        http://lore.ece.utexas.edu/~bgrayson/xosview.html

        The various forissier's kernel patches are at: http://www-
        isia.cma.fr/~forissie/smp_kernel_patch/


  3. HHooww ccaann II pprrooggrraamm ttoo uussee ttwwoo ((oorr mmoorree CCPPUUss)) ??

     Use a kernel-thread library. A good library, the pthread library
     made by Xavier Leroy
     http://pauillac.inria.fr/~xleroy/linuxthreads/>.

     LinuxThread is now integrated with glibc2 (aka libc6).

     From JJaakkoobb OOeesstteerrggaaaarrdd: Also consider using MPI. It's the industry
     standard message passing interface. It doesn't give you shared
     memory like threads, but it allows you to use your program in a
     cluster too.


  4. WWhhaatt hhaass cchhaannggeedd iinn tthhee tthhrreeaaddss ppaacckkaaggeess,, lliinnuuxxtthhrreeaadd,, eettcc..

     Glibc is the big change. glibc is threadsafe and includes
     linuxthreads Posix.4 threads by default. Real time signals are also
     in glibc so POSIX AIO should also be in glibc2.1 (I hope).


  5. HHooww ccaann II eennaabbllee mmoorree tthhaann 11 pprroocceessss ffoorr mmyy kkeerrnneell ccoommppiillee??

     use:

     ___________________________________________________________________
             # make [modules|zImage|bzImages] MAKE="make -jX"
             where X=max number of processes.
             WARNING: This won't work for "make dep".
     ___________________________________________________________________



  With a 2.1.x like kernel, see also the file
  /usr/src/linux/Documentation/smp for specific instruction.

  BTW, since running multiple compilers allows a machine with sufficient
  memory to use use the otherwise wasted CPU time during I/O caused
  delays make MAKE="make -j 2" -j 2 actually even helps on uniprocessor
  boxes (from RRaallff BBcchhllee).


  6. WWhhyy tthhee ttiimmee ggiivveenn bbyy tthhee time command is false ?  (from JJooeell
     MMaarrcchhaanndd)

     In the 2.0 series, the result given by the time command is false.
     The sum user+system is right *but* the spreading between user and
     system time is false.

     This bug in corrected in 2.1 series.


  7. HHooww wwiillll mmyy aapppplliiccaattiioonn ppeerrffoorrmm uunnddeerr SSMMPP??

     Look at SMP Performance of Linux
     http://www.interlog.com/~mackin/linux-smp.html> which gives useful
     hints how to bench a specific machine (from a post made by CCaammeerroonn
     MMaaccKKiinnnnoonn).


  8. WWhheerree ccaann II ffoouunndd mmoorree iinnffoorrmmaattiioonn aabboouutt ppaarraalllleell pprrooggrraammmmiinngg??

     Look at the Linux Parallel Processing HOWTO
     http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

     Lots of useful information can be found at Parallel Processing
     using Linux http://yara.ecn.purdue.edu/~pplinux/>




  33..  IInntteell aarrcchhiitteeccttuurree ssppeecciiffiicc qquueessttiioonnss


  33..11..  WWhhyy iitt ddooeessnn''tt wwoorrkk oonn mmyy mmaacchhiinnee??


  1. CCaann II uussee mmyy CCyyrriixx//AAMMDD//nnoonn--IInntteell CCPPUU iinn SSMMPP??

     SShhoorrtt aannsswweerr:: no.

     LLoonngg aannsswweerr:: Intel claims ownership to the APIC SMP scheme, and
     unless a company licenses it from Intel they may not use it. There
     are currently no companies that have done so.  (This of course can
     change in the future) FYI - Both Cyrix and AMD support the non-
     proprietary OpenPIC SMP standard but currently there are no
     motherboards that use it.


  2. WWhhyy ddooeessnn''tt mmyy oolldd CCoommppaaqq wwoorrkk??

     Put it into MP1.1/1.4 compliant mode.


  3. WWhhyy ddooeessnntt mmyy AALLRR wwoorrkk??

     From RRoobbeerrtt HHyyaatttt : ALR Revolution quad-6 seems quite safe, while
     some older revolution quad machines without P6 processors seem
     "iffy"...



  4. WWhhyy ddooeess SSMMPP ggoo ssoo sslloowwllyy?? or WWhhyy ddooeess oonnee CCPPUU sshhooww aa vveerryy llooww
     bbooggoommiippss vvaalluuee wwhhiillee tthhee ffiirrsstt oonnee iiss nnoorrmmaall??

     From AAllaann CCooxx: If one of your CPU's is reporting a very low
     bogomips value the cache is not enabled on it. Your vendor probably
     provides a buggy BIOS. Get the patch to work around this or better
     yet send it back and buy a board from a competent supplier.


  5. II''vvee hheeaarrdd IIBBMM mmaacchhiinneess hhaavvee pprroobblleemmss


     Some IBM machines have the MP1.4 bios block in the EBDA, allowed
     but not supported by < 2.1.80. Please update to the right kernel.

     There is an old 486SLC based IBM SMP box. Linux/SMP requires
     hardware FPU support.


  6. IIss tthheerree aannyy aaddvvaannttaaggee ooff IInntteell MMPP 11..44 oovveerr 11..11 ssppeecciiffiiccaattiioonn??

     Nope (according to Alan :) ), 1.4 is just a stricker specs of 1.1.


  7. WWhhyy ddooeess tthhee cclloocckk ddrriifftt ssoo rraappiiddllyy wwhheenn II rruunn lliinnuuxx SSMMPP??


     This is known problem with IRQ handling and long kernel locks in
     the 2.0 series kernels.  Consider upgrading to a later 2.1 kernel
     (not garenteed to work).

     From JJaakkoobb OOeesstteerrggaaaarrdd: Or, consider running xntpd. That should
     keep your clock right on time.  (I think that I've heard that
     enabling RTC in the kernel also fixes the clock drift. It works for
     me! but I'm not sure whether that's general or I'm just being
     lucky)



  8. WWhhyy aarree mmyy CCPPUU''ss nnuummbbeerreedd 00 aanndd 22 iinnsstteeaadd ooff 00 aanndd 11 ((oorr ssoommee ootthheerr
     oodddd nnuummbbeerriinngg))??

     The CPU number is assigned by the MB manufacturer and doesn't mean
     anything.  Ignore it.



  9. MMyy SSMMPP ssyysstteemm iiss lloocckkiinngg uupp aallll tthhee ttiimmee..  BBllaacckk ssccrreeeenn,, nnootthhiinngg iinn
     tthhee llooggss..  HHeellpp!!

     If you're running a 2.0 kernel, consider upgrading to later 2.0.32+
     kernels or apply Leonard Zubkoff's deadlock patch.  If you still
     have deadlocks, apply Ingo Molnar's deadlock detection patch and
     post the results (against your System.map) to linux-smp or linux-
     kernel.  You might also consider running a 2.1 kernel.





  33..22..  PPoossssiibbllee ccaauusseess ooff ccrraasshh

  You'll find in this section some ppoossssiibbllee reasons for a crash of an
  SMP machine (credits are due to JJaakkoobb OOeesstteerrggaaaarrdd for this part). As
  far as I (david) know, theses problems are Intel specific.

  +o  CCoooolliinngg pprroobblleemmss

     From RRaallff BBcchhllee: [Related to case size and fans] It's important
     that the air is flowing.  It of course can't where cables etc. are
     preventing this like in too small cases.  On the other side I've
     seen oversized cases causing big problems.  There are some tower
     cases on the market that actually are worse for cooling than
     desktops.  In short, the right thing is thinking about aerodynamics
     in the case.  Extra cases for hot peripherals are usefull as well.



  +o  BBaadd mmeemmoorryy

     Don't buy too cheap RAM and don't use mixed RAM modules on a
     motherboard that is picky about it.

     Especially Tyan motherboards are known to be picky about RAM speed.


  +o  BBaadd ccoommbbiinnaattiioonn ooff ddiiffffeerreenntt sstteeppppiinngg CCPPUUss

     Check /proc/cpuinfo to see that your CPUs are same stepping.


  +o  YYoouu aarree rruunnnniinngg 22..00..3333 aarreenn''tt yyoouu ??

     If you run 2.0.31 or 2.1.xx you can't be sure that SMP is stable.
     2.0.33 is the right kernel for a production system. 2.1.xx kernels
     perform better, but they are development releases and should NOT be
     considered stable!


  +o  IIff yyoouurr ssyysstteemm iiss uunnssttaabbllee,, tthheenn DDOONN''TT oovveerrcclloocckk iitt!!

     ...and even if it is stable, DON'T overclock.

     From RRaallff BBcchhllee: Overclocking causes very subtile problems.  I have
     a nice example, one of my overclocked old machines misscomputes a
     couple of pixels of a 640 x 400 fractal.  The problem is only
     visible when comparing them using tools. So better say _n_e_v_e_r_,
     _n_u_n_c_a_s_, _j_a_m_a_i_s_, _n_i_e_m_a_l_s overclock.



  +o  22..00..xx kkeerrnneell aanndd ffaasstt eetthheerrnneett (from RRoobbeerrtt GG.. BBrroowwnn)

     2.0.X kernels on high performance fast ethernet systems have
     significant (and known) problems with a race/deadlock condition in
     the networking interrupt handler.

     The solution is to get the latest 100BT development drivers from
     CESDIS (ones that define SMPCHECK).


  +o  AA bbuugg iinn tthhee 444400FFXX cchhiippsseett (from EEmmiill BBrriiggggss)

     If you had a system using the 440FX chipset then your problem with
     the lockups was possibly due to a documented errata in the chipset.
     Here is a reference

     References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX)
     Specification Update.  pg. 13

     http://www.intel.com/design/pcisets/specupdt/297654.htm

     The problem can be fixed with a bios workaround (Or a kernel patch)
     and in fact David Wragg wrote a patch that's included with Richard
     Gooch's mttr patch. For more information and a fix look here.

     http://nemo.physics.ncsu.edu/~briggs/vfix.html



  Some hardware is also known to cause problems. This includes:

  +o  AAddaapptteecc SSCCSSII ccoonnttrroolllleerrss

     Don't buy them, Adaptec is unsupportive to the linux developers.
     This is not a SMP problem, but a general high-performance Linux
     problem.


     It also seems that aic7xxx driver is broken under SMP (from RRoobbeerrtt
     HHyyaatttt).


     (from DDoouugg LLeeddffoorrdd, author of the Adaptec driver) Just a quick
     note, the 5.0.11 version of my driver for 2.0.33 is the one I
     [Doug] personally recommend for SMP and/or PII systems.  It's what
     I use here on a PII/266 dual system, although I'm running 2.1.92
     right now instead of 2.0.33.  Second note, the patch will not go
     into 2.0.34-pre6 cleanly, but can be used, and it has not been
     submitted for any of the 34pre kernels because I don't think it's
     had enough testing yet.


  +o  33CCoomm 33cc990055 ccaarrddss

     Some work, some don't. Try disabling busmastering if your system is
     unstable.




  33..33..  MMootthheerrbbooaarrdd ssppeecciiffiicc iinnffoorrmmaattiioonn

  Some more specific information can be found with the survey of SMP
  motherboards http://styx.phy.vanderbilt.edu/smp/mainboards.html>


  33..33..11..  MMootthheerrbbooaarrddss wwiitthh kknnoowwnn pprroobblleemmss


  +o  Gigabyte

     Solution: BIOS upgrade


  +o  SuperMicro

     Solution: BIOS upgrade


  +o  EPoX KP6-LS (CChhrriissttoopphheerr AAlllleenn WWiinngg, 16 march 1998)

     It appears to have the same BIOS related BogoMIPS problem as other
     motherboards. (one CPU only gives about 3 BogoMIPS, the other gives
     the full amount) All 2.0.x kernels lock up soon after booting, late
     2.1.x kernels run slowly but don't seem to lock up.  There is no
     BIOS upgrade available (yet). I wrote the manufacturer but have not
     received a reply.
  +o  Tyan

     Tyan motherboards are known to be picky about RAM speed (JJaakkoobb
     OOeesstteerrggaaaarrdd).

     From DDoouugg LLeeddffoorrdd about the onboard aic-7895 SCSI controller (for
     which he wrote the driver): "BTW, make sure you have at least BIOS
     version 1.16 on that Tyan motherboard.  The 1.15 and below BIOS
     versions have a bug relating to IRQ allocation for the 7895 SCSI
     controller" (submitted by SSzzaakkaaccssiittss SSzzaabboollccss).


  +o  GA686DLX (AAnnddrreeww CCrraannee)

     Same BIOS related BogoMIPS problem as other motherboards.

     Solution from Alan Cox: Congratulations, send the bill for your
     hair damage to the supplier. You have yet another SMP box with
     faulty bios. There is a patch for 2.0.x on www.uk.linux.org and
     there are people working on generic MTRR handling for 2.1.x


  +o  MS-6114

     More details for this motherboard at
     http://www.msi.com.tw/product/6114/6114.htm
     http://www.msi.com.tw/product/6114/6114.htm>

     Solution: BIOS upgrade

     Somebody experienced solid hangs (nothing in the log files) under
     constant load of about 5 running processes within less than 12
     hours with AMI BIOS v1.1.  v1.4b3 runs without problems.





  33..33..22..  MMootthheerrbbooaarrddss wwiitthh nnoo kknnoowwnn pprroobblleemmss


  +o  AIR P6NDP and P6NDI (LLeeoonnaarrdd NN.. ZZuubbkkooffff)

     My primary production machine is based on an AIR P6NDP and one of
     my test machines uses a P6NDI.  Both seem to be fine motherboards
     in my experience.  The P6NDI BIOS is a little conservative in its
     programming of the Natoma chipset for 50ns EDO, but a minor tweek
     to one register in rc.local took care of that.


  +o  AIR 54CDP (CChhrriiss MMaauurriittzz)

     You can also list the following motherboard as working with no
     problems:

     AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 /
     Redhat 5.0 (2.0.32 and 2.0.33 kernels)


  +o  HP XU 6/200 (JJeeaann--FFrraannccooiiss MMiiccoouulleeaauu)

     Works with 2.0 and 2.1 kernels. Some problems under high network
     load with 2.0.x kernel. Works under 2.1.78 with Ingo Molnar IO-APIC
     patch.


  +o  Elitegroup P6FX2-A (BBeenneeddiikktt HHeeiinneenn)

     Had this mainboard running with ONE PPro on it for several months,
     and since about a year, it's running without problems with TWO PPro
     200MHz. The only crashes this machine ever experienced were before
     Leonard Zubkoff's deadlock-patches for Linux 2.0.30... ;)

     Elitegroup P6FX2-A / ISA/PCI / Dual PPro200 / Debian "hamm"




  44..  UUsseeffuull ppooiinntteerrss


  44..11..  VVaarriioouuss


  +o  Parallel Processing using Linux
     http://yara.ecn.purdue.edu/~pplinux/>

  +o  Linux Parallel Processing HOWTO
     http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

  +o  ((oouuttddaatteedd)) Linux SMP home page
     http://www.uk.linux.org/SMP/title.html>

  +o  linux-smp mailing list

     To ssuubbssccrriibbee, send subscribe linux-smp in the message body at
     majordomo@vger.rutgers.edu

     To uunnssuubbssccrriibbee, send unsubscribe linux-smp in the message body at
     majordomo@vger.rutgers.edu


  +o  pthread library made by Xavier Leroy
     http://pauillac.inria.fr/~xleroy/linuxthreads/>

  +o  Linux SMP archives http://www.linuxhq.com/lnxlists/linux-smp/>

  +o  Survey of SMP motherboards
     http://styx.phy.vanderbilt.edu/smp/mainboards.html>

  +o  procps http://www.cs.inf.ethz.ch/~rauch/procps.html>

  +o  xosview http://lore.ece.utexas.edu/~bgrayson/xosview.html>

  +o  Pentium Pro Optimized BLAS and FFTs for Intel Linux
     http://www.cs.utk.edu/~ghenry/distrib/>

  +o  SMP Performance of Linux http://www.interlog.com/~mackin/linux-
     smp.html>

  +o  Multithreaded programs on linux
     http://www.e.kth.se/~e94_bek/mthread.html>




  44..22..  SSMMPP ssppeecciiffiicc ppaattcchheess


  +o  Forissier kernel patches http://www-
     isia.cma.fr/~forissie/smp_kernel_patch/>

  +o  Patch for a bug in the 440FX chipset
     http://nemo.physics.ncsu.edu/~briggs/vfix.html>

  +o  MTRR patch (latest version: 1.9)
     http://www.atnf.csiro.au/~rgooch/kernel-patches.html>



  55..  GGlloossssaarryy


  +o  SSMMPP Symetric Multi-Processors

  +o  AAPPIICC Advanced Programmable Interrupt Controler

  +o  tthhrreeaadd A thread is a processor activity in a process. The same
     process can have multiple threads. Those threads share the process
     address space and can therefore share data.

  +o  pptthhrreeaadd Posix thread, threads defined by the Posix standard.



  66..  LLiisstt ooff ccoonnttrriibbuuttoorrss

  Many thanks to those who help me to maintain this FAQ.


  +o  Emil Briggs

  +o  Robert G. Brown

  +o  Samuel S. Chessman

  +o  Alan Cox

  +o  Andrew Crane

  +o  Jocelyne Erhel

  +o  Byron Faber

  +o  Benedikt Heinen

  +o  Robert Hyatt

  +o  Tony Kocurko

  +o  Doug Ledford

  +o  Cameron MacKinnon

  +o  Joel Marchand

  +o  Chris Mauritz

  +o  Jean-Francois Micouleau

  +o  Jakob Oestergaard

  +o  Jean-Michel Rouet

  +o  Ralf Bchle

  +o  Sumit Roy

  +o  Szakacsits Szabolcs

  +o  El Warren

  +o  Christopher Allen Wing

  +o  Leonard N. Zubkoff



























































