Mail2News Mini-Howto <author>Rick Dean <tt><howto@fdd.com></tt> <date>v2.1, 1 September 1999 <abstract> This document describes how to feed a mailing list to a news server. </abstract>   <sect>Copyright, Distribution, etc. <sect1>What is this? This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO. <sect1>Copyright and such Copyright (c) 1999, Rick Dean. <newline> Copyright (c) 1996, Robert Hart. The authors retain their copyright of this document. You are hereby granted permission to redistribute this document in whole or in part as long as it includes this copyright notice. Commercial redistribution is allowed and encouraged. All translations or derivative works of this document must be covered under this copyright notice, and without additional restrictions on distribution. This arrangement is also known as a copyleft. This copyright notice, itself, is hereby placed in the public domain. You may copy it without atribution. <sect1>Where can I get this HOW-TO? The latest version of this document is available at <url url="http://fdd.com/howto/" name="http://fdd.com/howto/">. Many Linux distributions include HOWTOs in the <tt>/usr/doc/HOWTO</tt> directory. This mini-HOWTO like most Linux mini-HOWTO's was written initially is SGML. This allows the text to be automatically translated to many formats including text, HTML, PostScript, etc. Those other formats are available somewhere. More information about Linux documentation can be found at the <url url="http://www.redhat.com/mirrors/LDP/" name="Linux Documentation Project">, and so many other places. <sect1>Document history <itemize> <item>Version 1.0 was written by Robert Hart <tt><hartr@redhat.com></tt> in 1996 <item>Versions 2.x were modified by Rick Dean <tt><howto@fdd.com></tt> starting in July 1999 </itemize> <sect>Overview <sect1>What? Why? and some definitions This document describes how to feed a mailing list to a news server using a Linux box. It is called a mini-HOWTO, specifically the Mail2news mini-HOWTO. A mailing list (also known as a remailer), is an address where e-mail will be resent to a list of other addresses. This is useful for colaboration of geographically disperse groups. Many standards bodies like the working groups of the IETF use mailing lists. Unfortunately, if one is subscribed to several mailing lists, one's inbox may be routinely flooded. Furthermore, some companies (such as 3Com) specify which e-mail client (such as Lotus Notes) their employees must suffer with. Redirecting these e-mails to a news server frees people to choose a news reader and utilize refined features specifically designed for the task (of deriving signal from noise). News servers started on the Internet long ago, many years before the WWW. They (and the news reader clients) have features such as... <itemize> <item>threading - responses are grouped with the e-mails they refer to <item>read indication - the computer keeps track of what you have seen (even across newsgroups!) so you don't have to. <item>subject kill - mark threads as read by thread. <item>archiving <item>segregation by newsgroup (i.e. mailing list) <item>a quick standardized method subscribing/unsubscribing <item>and more.. </itemize> A big focus of news servers is sharing news between servers. The largest of these groups became known as USENET. This mini-HOWTO does not address that. You could share the newsgroups created with this mini-HOWTO on your own, but you will live just fine without it. Like a web server, ubiquitous Internet connectivity has made centralized news servers acceptable. Furthermore, recent benchmarks have shown single-processor Linux boxes can handle 1300+ HTTP hits per second, so scalability is a minor issue. Athough you do not need to own the mailing list to use mail2news, it is a good idea to own the news server. <sect1>Assumed environment This document assumes you are using Linux, but other Unices are nice. Currently only sendmail (for a mail delivery agent) is described, but as <tt>qmail</tt> grows in popularity (in part because it is easier to configure). Hopefully someone with submit configuration notes for it too. For a news server, this document describes <tt>innd</tt>. It is pretty dominant as news servers go, but any NNTP compliant one should work. A bit of glue called mail2news.pl is a perl script, thus you need the Perl interpreter, but it is very common and probably already installed. Finally, I assume you are running all this (except the mailing list remailer) on one machine. Dividing it up is left as an exercise for the reader. :-) At the time of this writing this mini-HOWTO was only tested against a RedHat-6.0 distribution. As a good computer scientist, you should not believe anything works until it has been specifically tested. Any feedback or notes relating to other distributions would be welcomed by the author. For most of this mini-HOWTO you will need root access unless otherwise specified. <sect1>Methodology This mini-HOWTO is presented backwards, as this is the easiest way to build and debug it. Backwards means we start with the newsreader and work upstream to the mailing list remailer, the opposite direction of normal data flow. This systems uses several hairy pieces (like sendmail and innd) which are sizeable mini-HOWTO's in thier own right. <sect1>Not covered This mini-HOWTO does not cover... <itemize> <item>setting up a mailing list remailer. <item>exchanging news between news servers. <item>a complete list of things not covered. </itemize> Please do not e-mail me about these subjects (or SPAM). <sect> The news reader <sect1>Netscape Netscape comes with an integrated news reader. The easy way to subscribe to a group is type in (or click on) a link like <tscreen><verb> news://news.sfour.com/ietf.confctrl </verb></tscreen> Once you have subscribed, you only need to go to the message center. This can be done by clicking on the small talk balloons icon in the bottom right of a browser window. <sect1>Free Agent Free Agent is a wonderful news reader by Forte for Windoze. See http://www.forteinc.com/agent/freagent.htm <sect1>trn and friends The old command line newsgreaders like <tt>trn</tt> and <tt>tin</tt> are a good standby. Be sure to set the environment variable NNTPSERVER first. For example... <tscreen><verb> export NNTPSERVER=news.sfour.com trn </verb></tscreen> <sect> The news server <sect1> Installing the news server You need to install a news server. I used <tt>inn</tt> but others are available (somewhere). Preferably just check the "News Server" box during your initial install, but alternatively if your distribution uses RPM (RedHat package manager), then use something like... <tscreen><verb> rpm -i inn-2.2.9.i386.rpm </verb></tscreen> <sect1> Running the newsserver To manually start or stop the news server, use a command like <tscreen><verb> /etc/rc.d/init.d/innd start </verb></tscreen> or <tscreen><verb> /etc/rc.d/init.d/innd stop </verb></tscreen> or <tscreen><verb> /etc/rc.d/init.d/innd restart </verb></tscreen> To have the news server start at boot you could add a command like this to end of <tt>/etc/rc.d/rc.local</tt>, but that is not the best. Many distributions have a graphical tool for choosing which daemons run. You can also try the command line program <tscreen><verb> setup </verb></tscreen> or <tscreen><verb> chkconfig --add innd </verb></tscreen> <sect1> Creating the newsgroup <tt>innd</tt> is pretty picky about permissions and ownership. For much of the news config you will need to be the user <tt>news</tt>. To become this user from root... <tscreen><verb> su - news </verb></tscreen> Using <tt>ctlinnd</tt>, create the newsgroup on your news server. Remember, the newsgroup will be <bf>local</bf>, so start it with a distinctive name so you can filter it out from your news distributions if you do that stuff. I shamelessly named my newsgroup <tt>ietf.confctrl</tt>. The words from left to right go from less to more specific. You also need to tell <tt/innd/ that the group is moderated (by using ctlinnd). Indicating a moderated group is done by specifying <tt>m</tt> to the <tt>newgroup</tt> command. For example... <tscreen><verb> ctlinnd newgroup ietf.confctrl m confctrl@isi.edu </verb></tscreen> The newsgroup is set up as a <bf/moderated/ group, as this allows us to take advantage of the email capabilities of innd. Any messages posted to a moderated group are <bf/not/ immediately submitted to the group. Instead, messages are emailed to the moderator of the group. In our example confctrl@isi.edu is the address which is resent by the remailer. If you are sharing news with other servers, remember to edit your newsfeeds so that this group is not <bf/not/ distributed (unless you specificaly wish this to occur). <sect1>Unrestricting access By default the news server, doesn't let any clients read news, so I needed to disable the user authentication of innd. This was done in the /etc/news/nnrp.access. Check out the nnrp.access man page to learn the syntax of this file. I changed the first non-comment line to ... <tscreen><verb> *:Read Post:::* </verb></tscreen> If you want a username/password, fill in the 3rd and 4th (colin separated) fields. For more information on the syntax, check the man page... <tscreen><verb> man nnrp.access </verb></tscreen> <sect1>Changing permissions of /usr/bin/rnews I had to change the permissions of /usr/bin/rnews. It was not world read/executable, but <tt>sendmail</tt> runs scripts as <tt>nobody</tt>. <tscreen><verb> chmod a+rx /usr/bin/rnews </verb></tscreen> <sect1>Testing article posting If you copy the following article to a file named <tt>rick.article</tt>... <code> Path: howto From: howto@fdd.com Message-ID: <199907120548.AAA05475@fdd.com> Subject: test Date: Mon, 12 Jul 1999 00:48:49 -0500 (CDT) Newsgroups: ietf.confctrl Approved: ietf-confctrl@kepler.hedland.edu.au NNTP-Posting-Host: localhost Organisation: (mail2news gateway) test </code> Then you should be able to post a file with... <tscreen><verb> /usr/bin/rnews -r localhost <rick.article </verb></tscreen> <sect1>Did it work? You don't need to wait for the article to show up as unread, just look at the <tt>/var/spool/news/articles/</tt> subdirectories for files being created. <sect1>Increasing expiration times You may want to increase the expiration time for articles of your new newsgroup. In my case I wanted them never to expire, so I added the following line.... <tscreen><verb> ietf*:A:never:never:never </verb></tscreen> ....to the <tt>/etc/news/expire.ctl</tt> file. To learn more about the syntax of this file type... <tscreen><verb> man expire.ctl </verb></tscreen> <sect> The posting script & perl <sect1>What is Perl? Perl stands for Practical Extension and Report Language. It is very popular for small scripts which manipulate text which is exactly what we need. Perl is installed by default on almost every Unix system. <sect1>Location of perl If you perl intepreter is in an unusual place (not <tt>/usr/bin/</tt>) then you will have to modify the first line of the script. If this line is wrong, on my 2.2 kernel system I get "bash: /usr/local/bin/mail2news.pl: No such file or directory" Can we please change this to "bash: /usr/local/bin/mail2news.pl: Interpreter not found. Check first line of script." ? <sect1>The mail2news.pl script <code> #!/usr/bin/perl ($program = $0) =~ s%.*/%%; #( $version ) = $] =~ /(\d+\.\d+).*\nPatch level/; #die "$program: requires at least version 3 of perl\n" # if $version < 3; # $news_poster_program = "/usr/bin/inews"; # $news_poster_options = "-h -o \"mail2news gateway\""; $news_poster_program = "/usr/bin/rnews"; $news_poster_options = "-r localhost"; $postinghost = "localhost"; if ($#ARGV < 0) { # $newsgroup = "test"; # we'll expect the newsgroup line in the body } elsif ($#ARGV == 0) { $newsgroup = $ARGV[0]; } else { die "usage: $program [newsgroup]\n"; } # in case inews dumps core or something crazy $SIG{'PIPE'} = "plumber"; sub plumber { die "$program: \"$news_poster_program\" died prematurely!\n"; } open (INEWS, "| $news_poster_program $news_poster_options") || die "$program: can't run $news_poster_program\n"; # header munging loop while (<STDIN>) { last if /^$/; # transform real from: line back to icky style s/^From:\s+(.*) <(.*)>/From: $2 ($1)/; s/Message-Id/Message-ID/; # transform from_ line to path header; also works locally s/^From\s+(\S+)@(\S+).*/Path: $2!$1/ || s/^From\s+(\S+)[^@]*$/Path: $1\n/; print INEWS # if /^(Date|From|Subject|Path|Newsgroups|Organization|Message-ID):/i; if /^(Date|From|Subject|Path|Newsgroups|Message-ID):/i; $saw_subject |= ( $+ eq 'Subject' ); $saw_msgid |= ( $+ eq 'Message-ID' ); # $saw_newsgroup |= ( $+ eq 'Newsgroups' ); } warn "$program: didn't expect newsgroup in both headers and ARGV\n" if $newsgroup && $saw_newsgroup; die "$program: didn't get newsgroup from either headers or ARGV\n" unless $newsgroup || $saw_newsgroup; $approved = $newsgroup; $approved =~ s/\./'-'/eg; ($sec,$min,$hour,$mday,$mon,$year)=localtime(time); $madeupid = "\<$year$mon$mday.$hour$min$sec.$$\@kepler.hedland.edu.au\>"; printf INEWS "Newsgroups: %s\n", $newsgroup if $newsgroup; printf INEWS "Approved: %s\@kepler.hedland.edu.au\n", $approved; print INEWS "Subject: Untitled\n" unless $saw_subject; printf INEWS "Message-ID: %s\n", $madeupid unless $saw_msgid; printf INEWS "NNTP-Posting-Host: %s\n", $postinghost; print INEWS "Organisation: (mail2news gateway)\n"; print INEWS "\n"; print INEWS while <STDIN>; # gobble rest of message close INEWS; exit $?; </code> I saved the script in /usr/local/bin (and will use this path throughout the HOWTO). Be sure to make the script executable by all, but not writable by group or other. Sendmail is picky. <tscreen><verb> chmod a+x /usr/local/bin/mail2news.pl chmod go-w /usr/local/bin/mail2news.pl </verb></tscreen> or <tscreen><verb> chmod 555 /usr/local/bin/mail2news.pl </verb></tscreen> for short. <sect1>How do I know if the script is running? I tested this script by changed my news poster from /usr/bin/rnews to /bin/cat. I then saved an e-mail send to myself in a file. Finally I ran the mail2news.pl on the saved mail and captured the output to a file. <tscreen><verb> /usr/local/bin/mail2news.pl ietf.confctrl </tmp/savedMailFile >/tmp/article </verb></tscreen> <sect1>What is with the Aussie? Yes, the output of your posting script should contain the e-mail address of an Austrailian. My guess is that his address is a trusted address in your news configuration (althogh I could not find it in mine) for approving moderated postings. <sect1>What do you mean $PATH? If the mail2news.pl script is not in my path, I get the error <tt>bash: mail2news.pl: command not found</tt>. You will need to either add this directory to your path <tscreen><verb> PATH=$PATH:/usr/local/bin </verb></tscreen> (which only works for the current login) or give an absolute path when you run the script <tscreen><verb> /usr/local/bin/mail2news.pl </verb></tscreen> To view your current path type <tscreen><verb> echo $PATH </verb></tscreen> <sect> The local mailer daemon <sect1>Overview We need to create a local e-mail address for the remailer to mail to. This address will be configured to run a program (to post the newsgroup) every time an e-mail arrives. Unfortunately, (for us) this feature (called "pipe to program") is not usually enabled for security reasons. This HOWTO only describes modifying sendmail. <tt>qmail</tt> is another excellent option, but not yet covered here. <sect1>Creating a mail alias Instead of creating a new user account, we will only create an alias. For sendmail these are kept in <tt>/etc/aliases</tt>. The syntax is defined in the man page... <tscreen><verb> man aliases </verb></tscreen> In our example the username of the mail address is <tt>confctrl</tt> which posts to the local newsgroup <tt>ietf.confctrl</tt>, so we add a line like ... <tscreen><verb> confctrl: "| /usr/local/bin/mail2news.pl ietf.confctrl " </verb></tscreen> The double quotes are required. There cannot be a space between the first double quotes and the | (pipe) character, or sendmail will complain "User unknown". Huh? Whenever you modify the <tt>/etc/aliases</tt> file you need to notify sendmail. <tscreen><verb> sendmail -bi </verb></tscreen> <sect1>Did it work? After you send an e-mail, check the /var/log/maillog to see if it worked. The log may contain a useful error message. I found it convenient to open up another terminal window to follow the log with <tscreen><verb> tail -f /var/log/maillog </verb></tscreen> <sect1>Enabling "pipe to program" "Pipe to program" is the CGI of mail. When an e-mail arrives, instead of appending to a file, sendmail runs a program (or script) and feeds the mail to its standard input. Actually, with RedHat-6.0 sendmail comes with "pipe to program" enabled, but essentially only a limited number of programs are runnable. User security was not an issue on my machine, so I found it easiest to enable running of everything. Besides, there was no man page for smrsh. To liberalize "pipe to program" we need to modify <tt>sendmail.cf</tt> (by modifying <tt> sendmail.mc</tt>) so that the restricted shell is not used. Strangely, uncommenting the smrsh feature didn't work, I needed to change the shell from <tt>/usr/sbin/smrsh</tt> to <tt>/bin/bash</tt>. Without this change I kept getting a <tt>"Service unavailable"</tt> error message in the <tt>/var/log/maillog</tt> file. The header of <tt>/etc/senmail.mc</tt> of RedHat-6 has a bug. The proper command line is... <tscreen><verb> m4 /etc/sendmail.mc >/etc/sendmail.cf </verb></tscreen> You need to do this when you change <tt>sendmail.mc</tt>. Hopefully, RedHat will extend the super cool Makefile idea in <tt>/etc/mail</tt> so you only have to type <tt>make</tt>. You will need to install <tt>sendmail-cf.</tt>something<tt>.rpm</tt> first. e.g. ... <tscreen><verb> rpm -i sendmail-cf-8.9.3-10.i386.rpm </verb></tscreen> Whenever you modify the sendmail.cf file, you should restart sendmail... <tscreen><verb> /etc/rc.d/init.d/sendmail restart </verb></tscreen> <sect1>Documentation With RedHat, sendmail documentation does not install by default. I comes in a separate RPM with a name like sendmail-doc-8.9.3-10.i386.rpm. This will create <tt>/usr/doc/sendmail</tt> <tscreen><verb> rpm -i sendmail-doc-* </verb></tscreen> http://www.sendmail.org/~ca/email/doc/op-sh-5.html describes the syntax of sendmail.cf. I hope you don't have to use it. This is a description of the features you are modifying in sendmail.mc http://www.sendmail.org/m4/features.html Sendmail has a man page... <tscreen><verb> man sendmail </verb></tscreen> <sect1>Unprivledged If you are having trouble, and create an e-mail alias with a different (simpler) target script to test it, remember that sendmail runs the program as an unprivledged user, who probably doesn't have privledges to write anywhere except globally writable directories such as <tt>/tmp</tt>. <sect> The mailing list remailer <sect1>Subscribing The method of subscribing to each e-mail list is different, although most involve sending some kind of e-mail message. Please be very careful not to send to the replicating address of the mailgroup. This is a sure way to annoy the members of the list. Most remailers (but not all) have a separate address for subscribing. In confctrl, the administration address is confctrl-request@isi.edu as explained by http://www.ietf.org/html.charters/mmusic-charter.html. When you successfully subscribe you should (hopefully) see a welcome message in the newsgroup. Some mailing lists do not let you subscribe an address you are not mailing from. In this case you will need to forge the return address of an e-mail. <sect1>Forging mail Forgeing mail is easiest to do with an old Netscape client. It would accept anything for a return address. Fortunately, modern browsers require you to retreive mail successfully for an address before they will let you send. Thus, you are required to fall back the tried and true method....telnet. <sect2>Looking up MX hosts When you speficy a web page, your browser does a DNS lookup to convert the domain name into an IP address. This is a lookup of an "A" record. (Also "CNAME" records are considered.) When sending mail a "MX" record is looked up. If this is missing a "CNAME" or "A" one is used. Thus, to forge mail you need to do an MX lookup. <tscreen><verb> dig mx isi.edu </verb></tscreen> will give... <tscreen><verb> ; <<>> DiG 8.2 <<>> mx isi.edu ;; res options: init recurs defnam dnsrch ;; got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 4 ;; QUERY SECTION: ;; isi.edu, type = MX, class = IN ;; ANSWER SECTION: isi.edu. 1D IN MX 0 tnt.isi.edu. isi.edu. 1D IN MX 10 venera.isi.edu. ;; AUTHORITY SECTION: isi.edu. 1D IN NS venera.isi.edu. isi.edu. 1D IN NS ns.isi.edu. isi.edu. 1D IN NS east.isi.edu. ;; ADDITIONAL SECTION: tnt.isi.edu. 1D IN A 128.9.128.128 venera.isi.edu. 1D IN A 128.9.176.32 ns.isi.edu. 1D IN A 128.9.128.127 east.isi.edu. 1D IN A 38.245.76.2 ;; Total query time: 448 msec ;; FROM: fdd.com to SERVER: default -- 127.0.0.1 ;; WHEN: Sun Jul 25 15:49:32 1999 ;; MSG SIZE sent: 25 rcvd: 182 </verb></tscreen> thus you would use tnt.isi.edu. (If you don't have dig, <tt>nslookup</tt> will do.) <sect2>SMTP Mail is delivered using the Simple Mail Transport Protocol (SMTP). Like most good Internet protocols, it is ASCII based to make troubleshooting and development easier. I will not explain everything, but simply give an example. Hopefully, this is enough. The protocol is line oriented. Each email as specified in RFC822, is composed of headers and body which are separated by the first blank line (no characters, not even spaces). SMTP specifies the end of an e-mail with a line containing only a period. <tscreen><verb> halyard$ telnet tnt.isi.edu 25 Trying 128.9.128.128... Connected to tnt.isi.edu. Escape character is '^]'. 220 tnt.isi.edu ESMTP Sendmail 8.8.7/8.8.6; Sun, 25 Jul 1999 14:01:25 -0700 (PDT) helo isi.edu 250 tnt.isi.edu Hello rick@node-d8e9822 [216.233.8.34] (may be forged), pleased to meet you mail from:<confctrl@news.sfour.com> 250 <confctrl@news.sfour.com>... Sender ok rcpt to:<confctrl-request@isi.edu> 250 <confctrl-request@isi.edu>... Recipient ok data 354 Enter mail, end with "." on a line by itself From:<confctrl@news.sfour.com> Subject: help help . 250 OAA26778 Message accepted for delivery quit 221 tnt.isi.edu closing connection Connection closed by foreign host. halyard$ </verb></tscreen> <sect1>Be patient The mailing list is last because in part it is the slowest. When you mail to the reflected address, it may take up to an hour to see the response. </article>