  Mail2News Mini-Howto
  Rick Dean <howto@fdd.com>
  v2.0, 1 July 1999

  This document describes how to feed a mailing list to a news server.

  1.  Copyright, Distribution, etc.

  1.1.  What is this?

  This document describes how to feed a mailing list to a news server
  using a Linux box.  It is called a mini-HOWTO, specifically the
  Mail2news mini-HOWTO.


  1.2.  Copyright and such

  Copyright (c) 1999, Rick Dean.
  Copyright (c) 1996, Robert Hart.


  The authors retain their copyright of this document.  You are hereby
  granted permission to redistribute this document in whole or in part
  as long as it includes this copyright notice.  Commercial
  redistribution is allowed and encouraged.  All translations or
  derivative works of this document must be covered under this copyright
  notice, and without additional restrictions on distribution.  This
  arrangement is also known as a copyleft.


  This copyright notice, itself, is hereby placed in the public domain.
  You may copy it without atribution.


  1.3.  Where can I get this HOW-TO?

  The latest version of this document is available at
  http://fdd.com/howto/ <http://fdd.com/howto/>.


  Many Linux distributions include mini-HOWTOs in the /usr/doc/HOWTO
  directory.


  This mini-HOWTO like most Linux mini-HOWTO's was written initially is
  SGML.  This allows the text to be automatically translated to many
  formats including text, HTML, PostScript, etc.  Those other formats
  are available somewhere.


  More information about Linux documentation can be found at the Linux
  Documentation Project <http://www.redhat.com/mirrors/LDP/>, and so
  many other places.


  1.4.  Document history


    Version 1.0 was written by Robert Hart <hartr@redhat.com> in 1996

    Version 2.0 was written by Rick Dean <howto@fdd.com> in 1999

  2.  Overview

  2.1.  What? Why? and some definitions

  This document describes how to feed a mailing list to a news server
  using a Linux box.  It is called a mini-HOWTO, specifically the
  Mail2news mini-HOWTO.


  A mailing list (also known as a remailer), is an address where e-mail
  will be resent to a list of other addresses.  This is useful for
  colaboration of geographically disperse groups.  Many standards bodies
  like the working groups of the IETF use mailing lists.


  Unfortunately, if one is subscribed to several mailing lists, one's
  inbox may be routinely flooded.  Furthermore, some companies (such as
  3Com) specify which e-mail client (such as Lotus Notes) their
  employees must suffer with.  Redirecting these e-mails to a news
  server frees people to choose a news reader and utilize refined
  features specifically designed for the task (of deriving signal from
  noise).


  News servers started on the Internet long ago, many years before the
  WWW.  They (and the news reader clients) have features such as...

    threading - responses are grouped with the e-mails they refer to

    read indication - the computer keeps track of what you have seen
     (even across newsgroups!) so you don't have to.

    subject kill - mark threads as read by thread.

    archiving

    segregation by newsgroup (i.e. mailing list)

    a quick standardized method subscribing/unsubscribing

    and more..


  A big focus of news servers is sharing news between servers.  The
  largest of these groups became known as USENET.  This mini-HOWTO does
  not address that.  You could share the newsgroups created with this
  mini-HOWTO on your own, but you will live just fine without it.  Like
  a web server, ubiquitous Internet connectivity has made centralized
  news servers acceptable.  Furthermore, recent benchmarks have shown
  single-processor Linux boxes can handle 1300+ HTTP hits per second, so
  scalability is a minor issue.


  Athough you do not need to own the mailing list to use mail2news, it
  is a good idea to own the news server.


  2.2.  Assumed environment

  This document assumes you are using Linux, but other Unices are nice.
  Currently only sendmail (for a mail delivery agent) is described, but
  as qmail grows in popularity (in part because it is easier to
  configure).  Hopefully someone with submit configuration notes for it
  too.  For a news server, this document describes innd.  It is pretty
  dominant as news servers go, but any NNTP compliant one should work.
  A bit of glue called mail2news.pl is a perl script, thus you need the
  Perl interpreter, but it is very common and probably already
  installed.  Finally, I assume you are running all this (except the
  mailing list remailer) on one machine.  Dividing it up is left as an
  exercise for the reader. :-)


  At the time of this writing this mini-HOWTO was only tested against a
  RedHat-6.0 distribution.  As a good computer scientist, you should not
  believe anything works until it has been specifically tested.  Any
  feedback or notes relating to other distributions would be welcomed by
  the author.


  For most of this mini-HOWTO you will need root access unless otherwise
  specified.


  2.3.  Methodology

  This mini-HOWTO is presented backwards, as this is the easiest way to
  build and debug it.  Backwards means we start with the newsreader and
  work upstream to the mailing list remailer, the opposite direction of
  normal data flow.  This systems uses several hairy pieces (like
  sendmail and innd) which are sizeable mini-HOWTO's in thier own right.


  2.4.  Not covered

  This mini-HOWTO does not cover...

    setting up a mailing list remailer.

    exchanging news between news servers.

    a complete list of things not covered.


  Please do not e-mail me about these subjects (or SPAM).


  3.  The news reader

  3.1.  Netscape

  Netscape comes with an integrated news reader.  The easy way to
  subscribe to a group is type in (or click on) a link like


       news://fuji.sfour.com/ietf.confctrl

  Once you have subscribed, you only need to go to the message center.
  This can be done by clicking on the small talk balloons icon in the
  bottom right of a browser window.


  3.2.  Free Agent

  Free Agent is a wonderful news reader by Forte for Windoze.  See
  http://www.forteinc.com/agent/freagent.htm

  3.3.  trn and friends

  The old command line newsgreaders like trn and tin are a good standby.
  Be sure to set the environment variable NNTPSERVER first.  For
  example...


       export NNTPSERVER=fuji.sfour.com
       trn

  4.  The news server

  4.1.  Installing the news server

  You need to install a news server.  I used inn but others are
  available (somewhere).


  Preferably just check the "News Server" box during your initial
  install, but alternatively if your distribution uses RPM (RedHat
  package manager), then use something like...


       rpm -i inn-2.2.9.i386.rpm

  4.2.  Running the newsserver

  To manually start or stop the news server, use a command like


       /etc/rc.d/init.d/innd start

  or


       /etc/rc.d/init.d/innd stop

  or


       /etc/rc.d/init.d/innd restart

  To have the news server start at boot you could add a command like
  this to end of /etc/rc.d/rc.local, but that is not the best.  Many
  distributions have a graphical tool for choosing which daemons run.
  You can also try the command line program


  setup

  or


       chkconfig --add innd

  4.3.  Creating the newsgroup

  innd is pretty picky about permissions and ownership.  For much of the
  news config you will need to be the user news.  To become this user
  from root...


               su - news

  Using ctlinnd, create the newsgroup on your news server. Remember,
  the newsgroup will be local, so start it with a distinctive name so
  you can filter it out from your news distributions if you do that
  stuff.  I shamelessly named my newsgroup ietf.confctrl.  The words
  from left to right go from less to more specific.


  You also need to tell innd that the group is moderated (by using
  ctlinnd).  Indicating a moderated group is done by specifying m to the
  newgroup command.  For example...


               ctlinnd newgroup ietf.confctrl m confctrl@isi.edu

  The newsgroup is set up as a moderated group, as this allows us to
  take advantage of the email capabilities of innd. Any messages posted
  to a moderated group are not immediately submitted to the group.
  Instead, messages are emailed to the moderator of the group.  In our
  example confctrl@isi.edu is the address which is resent by the
  remailer.


  If you are sharing news with other servers, remember to edit your
  newsfeeds so that this group is not not distributed (unless you
  specificaly wish this to occur).


  4.4.  Unrestricting access

  By default the news server, doesn't let any clients read news, so I
  needed to disable the user authentication of innd.  This was done in
  the /etc/news/nnrp.access.  Check out the nnrp.access man page to
  learn the syntax of this file.  I changed the first non-comment line
  to ...

  *:Read Post:::*

  If you want a username/password, fill in the 3rd and 4th (colin
  separated) fields.  For more information on the syntax, check the man
  page...


       man nnrp.access

  4.5.  Changing permissions of /usr/bin/rnews

  I had to change the permissions of /usr/bin/rnews.  It was not world
  read/executable, but sendmail runs scripts as nobody.


       chmod a+rx /usr/bin/rnews

  4.6.  Testing article posting

  If you copy the following article to a file named rick.article...

  ______________________________________________________________________
  Path: rick
  From: rick@fdd.com
  Message-ID: <199907120548.AAA05475@fdd.com>
  Subject: test
  Date: Mon, 12 Jul 1999 00:48:49 -0500 (CDT)
  Newsgroups: ietf.confctrl
  Approved: ietf-confctrl@kepler.hedland.edu.au
  NNTP-Posting-Host: localhost
  Organisation: (mail2news gateway)


  test
  ______________________________________________________________________

  Then you should be able to post a file with...


       /usr/bin/rnews -r localhost <rick.article

  4.7.  Did it work?

  You don't need to wait for the article to show up as unread, just look
  at the /var/spool/news/articles/ subdirectories for files being
  created.
  4.8.  Increasing expiration times

  You may want to increase the expiration time for articles of your new
  newsgroup.  In my case I wanted them never to expire, so I added the
  following line....


       ietf*:A:never:never:never

  ....to the /etc/news/expire.ctl file.


  To learn more about the syntax of this file type...


       man expire.ctl

  5.  The posting script & perl

  5.1.  What is Perl?

  Perl stands for Practical Extension and Report Language.  It is very
  popular for small scripts which manipulate text which is exactly what
  we need.


  Perl is installed by default on almost every Unix system.


  5.2.  Location of perl

  If you perl intepreter is in an unusual place (not /usr/bin/) then you
  will have to modify the first line of the script.  If this line is
  wrong, on my 2.2 kernel system I get "bash:
  /usr/local/bin/mail2news.pl: No such file or directory" Can we please
  change this to "bash: /usr/local/bin/mail2news.pl: Interpreter not
  found.  Check first line of script." ?


  5.3.  The mail2news.pl script

  ______________________________________________________________________
  #!/usr/bin/perl

  ($program = $0) =~ s%.*/%%;

  #( $version  ) = $] =~ /(\d+\.\d+).*\nPatch level/;
  #die "$program: requires at least version 3 of perl\n"
  #        if $version < 3;

  # $news_poster_program = "/usr/bin/inews";
  # $news_poster_options = "-h -o \"mail2news gateway\"";
  $news_poster_program = "/usr/bin/rnews";
  $news_poster_options = "-r localhost";
  $postinghost = "localhost";

  if ($#ARGV < 0) {
      # $newsgroup = "test";
      # we'll expect the newsgroup line in the body
  } elsif ($#ARGV == 0) {
      $newsgroup = $ARGV[0];
  } else {
      die "usage: $program [newsgroup]\n";
  }

  # in case inews dumps core or something crazy
  $SIG{'PIPE'} = "plumber";
  sub plumber { die "$program: \"$news_poster_program\" died prematurely!\n"; }

  open (INEWS, "| $news_poster_program $news_poster_options") ||
      die "$program: can't run $news_poster_program\n";

  # header munging loop
  while (<STDIN>) {
     last if /^$/;

     # transform real from: line back to icky style
     s/^From:\s+(.*) <(.*)>/From: $2 ($1)/;

     s/Message-Id/Message-ID/;

     # transform from_ line to path header; also works locally
     s/^From\s+(\S+)@(\S+).*/Path: $2!$1/
       || s/^From\s+(\S+)[^@]*$/Path: $1\n/;

     print INEWS
  #       if /^(Date|From|Subject|Path|Newsgroups|Organization|Message-ID):/i;
     if /^(Date|From|Subject|Path|Newsgroups|Message-ID):/i;
     $saw_subject |= ( $+ eq 'Subject' );

     $saw_msgid |= ( $+ eq 'Message-ID' );

  #   $saw_newsgroup |= ( $+ eq 'Newsgroups' );
  }

  warn "$program: didn't expect newsgroup in both headers and ARGV\n"
      if $newsgroup && $saw_newsgroup;

  die "$program: didn't get newsgroup from either headers or ARGV\n"
      unless $newsgroup || $saw_newsgroup;

  $approved = $newsgroup;
  $approved =~ s/\./'-'/eg;

  ($sec,$min,$hour,$mday,$mon,$year)=localtime(time);
  $madeupid = "\<$year$mon$mday.$hour$min$sec.$$\@kepler.hedland.edu.au\>";

  printf INEWS "Newsgroups: %s\n", $newsgroup if $newsgroup;
  printf INEWS "Approved: %s\@kepler.hedland.edu.au\n", $approved;
  print  INEWS "Subject: Untitled\n" unless $saw_subject;
  printf INEWS "Message-ID: %s\n", $madeupid unless $saw_msgid;
  printf INEWS "NNTP-Posting-Host: %s\n", $postinghost;
  print  INEWS "Organisation: (mail2news gateway)\n";
  print  INEWS "\n";

  print INEWS while <STDIN>;   # gobble rest of message

  close INEWS;
  exit $?;
  ______________________________________________________________________

  I saved the script in /usr/local/bin (and will use this path
  throughout the HOWTO).


  Be sure to make the script executable by all, but not writable by
  group or other.  Sendmail is picky.


       chmod a+x /usr/local/bin/mail2news.pl
       chmod go-w /usr/local/bin/mail2news.pl

  or


       chmod 555 /usr/local/bin/mail2news.pl

  for short.


  5.4.  How do I know if the script is running?

  I tested this script by changed my news poster from /usr/bin/rnews to
  /bin/cat.  I then saved an e-mail send to myself in a file.  Finally I
  ran the mail2news.pl on the saved mail and captured the output to a
  file.

       /usr/local/bin/mail2news.pl ietf.confctrl </tmp/savedMailFile >/tmp/article

  5.5.  What is with the Aussie?

  Yes, the output of your posting script should contain the e-mail
  address of an austrailian.  My guess is that the Austrailian's address
  is trusted address in your news configuration (althogh I could not
  find it in mine) for approving moderated postings.

  5.6.  What do you mean $PATH?

  If the mail2news.pl script is not in my path, I get the error bash:
  mail2news.pl: command not found.  You will need to either add this
  directory to your path


       PATH=$PATH:/usr/local/bin

  (which only works for the current login) or give an absolute path when
  you run the script


       /usr/local/bin/mail2news.pl

  To view your current path type


       echo $PATH

  6.  The local mailer daemon

  6.1.  Enabling alias files

  This document only describes modifying sendmail.  Many people prefer
  qmail.


  Out of the box, sendmail does not come with the "pipe to program"
  feature enabled.  Rightly so, this feature is a security risk.  It
  enambles users to have their mail fed to a program instead of appended
  to a file.  (Users can configure this in thier .forward file.)


  Please don't believe me.  Test your own system first.  Much of this
  chapter might not be needed by you.


  6.2.  Documentation

  IMHO, sendmail installs with woefully insufficient documentation.


  http://www.sendmail.org/ ca/email/doc/op-sh-5.html describes the
  syntax of sendmail.cf. I hope you don't have to use it.


  This is a description of the features you are trashing in sendmail.mc
  http://www.sendmail.org/m4/features.html


  Sendmail has a man page...


       man sendmail

  6.3.  Recompiling your sendmail.cf file

  To get the "pipe to program" stuff in the aliases file to work you
  need to modify the default sendmail.cf (by modifying the sendmail.mc)
  so that the restricted shell is not used.  I suppose the proper
  solution would be to add the one program to the restricted shell list,
  but their was no man page on smrsh.  Strangely, uncommenting the smrsh
  feature didn't work, I needed to change the shell from /usr/sbin/smrsh
  to /bin/bash.  Yeah, this is slightly risky, but it was not an issue
  on my machine.  Without this change I kept getting a "Service
  unavailable" error message in the /var/log/maillog file.


  The header of /etc/senmail.mc of RedHat-6 has a bug.  The proper
  command line is...


       m4 /etc/sendmail.mc >/etc/sendmail.cf

  You need to do this when you change sendmail.mc.  Hopefully, RedHat
  will extend the super cool Makefile idea in /etc/mail You will need to
  install sendmail-cf.something.rpm first.  e.g. ...


       rpm -i sendmail-cf-8.9.3-10.i386.rpm

  Whenever you modify the sendmail.cf file, you should restart
  sendmail...


       /etc/rc.d/init.d/sendmail restart

  6.4.  Creating a mail alias

  Instead of creating a new user account, we will only create an alias.


  When modifying the /etc/aliases file, the double quotes are required.
  There cannot be a space between the first double quotes and the |
  (pipe) character, or sendmail will complain  "User unknown"


  Add a line like ...


       confctrl:   "| /usr/local/bin/mail2news.pl ietf.confctrl "

  Whenever you modify the /etc/aliases file you need to notify sendmail.

  sendmail -bi

  6.5.  Did it work?

  Check the /var/log/maillog to see if it worked, or for error messages.
  I found it useful to open up another terminal (ssh) window to watch
  the log with


       tail -f /var/log/maillog

  6.6.  Unprivledged

  If you are having trouble, and create an e-mail alias with a different
  (simpler) target program to test it, remember that sendmail runs the
  program as an unprivledged user, who probably doesn't have privledges
  to write anywhere except globally writable directories such as /tmp.


  7.  The mailing list remailer

  7.1.  Subscribing

  The method of subscribing to each e-mail list is different, although
  most involve sending some kind of e-mail message.  Please be very
  careful not to send to the replicating address of the mailgroup.  This
  is a sure way to annoy the members of the list.  Most remailers (but
  not all) have a separate address for subscribing.  In confctrl, the
  administration address is confctrl-request@isi.edu


  Some mailing lists do not let you subscribe an address you are not
  mailing from.  In this case you will need to forge the return address
  of an e-mail.


  When you successfully subscribe you should (hopefully) see a welcome
  message in the newsgroup.


  7.2.  Forging mail

  Forgeing mail is easiest to do with an old Netscape client.  It would
  accept anything for a return address.  Fortunately, modern browsers
  require you to retreive mail successfully for an address before they
  will let you send.  Thus you are required to fall back the tried and
  true method....telnet.


  7.2.1.  Looking up MX hosts

  When you speficy a web page, your browser does a DNS lookup to convert
  the domain name into an IP address.  This is a lookup of an "A"
  record.  (Also "CNAME" records are considered.)  When sending mail a
  "MX" record is looked up.  If this is missing a "CNAME" or "A" one is
  used.  Thus, to forge mail you need to do an MX lookup.


  dig mx isi.edu

  will give...


       ; <<>> DiG 8.2 <<>> mx isi.edu
       ;; res options: init recurs defnam dnsrch
       ;; got answer:
       ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6
       ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4
       ;; QUERY SECTION:
       ;;      isi.edu, type = MX, class = IN

       ;; ANSWER SECTION:
       isi.edu.                1D IN MX        0 tnt.isi.edu.
       isi.edu.                1D IN MX        10 venera.isi.edu.

       ;; AUTHORITY SECTION:
       isi.edu.                1D IN NS        venera.isi.edu.
       isi.edu.                1D IN NS        ns.isi.edu.
       isi.edu.                1D IN NS        east.isi.edu.

       ;; ADDITIONAL SECTION:
       tnt.isi.edu.            1D IN A         128.9.128.128
       venera.isi.edu.         1D IN A         128.9.176.32
       ns.isi.edu.             1D IN A         128.9.128.127
       east.isi.edu.           1D IN A         38.245.76.2

       ;; Total query time: 448 msec
       ;; FROM: fdd.com to SERVER: default -- 127.0.0.1
       ;; WHEN: Sun Jul 25 15:49:32 1999
       ;; MSG SIZE  sent: 25  rcvd: 182

  thus you would use tnt.isi.edu.


  7.2.2.  SMTP

  Mail is delivered using the Simple Mail Transport Protocol (SMTP).
  Like most good Internet protocols, it is ASCII based to make
  troubleshooting and development easier.  I will not explain
  everything, but simply give an example.  Hopefully, this is enough.


  The protocol is line oriented.  Each email as specified in RFC822, is
  composed of headers and body which are separated by the first blank
  line (no characters, not even spaces).  SMTP specifies the end of an
  e-mail with a line containing only a period.

  halyard$ telnet tnt.isi.edu 25
  Trying 128.9.128.128...
  Connected to tnt.isi.edu.
  Escape character is '^]'.
  220 tnt.isi.edu ESMTP Sendmail 8.8.7/8.8.6; Sun, 25 Jul 1999 14:01:25 -0700 (PDT)
  helo isi.edu
  250 tnt.isi.edu Hello rick@node-d8e9822 [216.233.8.34] (may be forged), pleased to meet you
  mail from:<confctrl@fuji.sfour.com>
  250 <confctrl@fuji.sfour.com>... Sender ok
  rcpt to:<confctrl-request@isi.edu>
  250 <confctrl-request@isi.edu>... Recipient ok
  data
  354 Enter mail, end with "." on a line by itself
  From:<confctrl@fuji.sfour.com>
  Subject: help

  help

