Configure the INN nntp server (2021/2022)
< Postfix configure (2021 edition) | Configuring a small home server in 2021 with openSUSE Leap 15.3 | MySQL - Mariadb 2021 >
Références
RFC
INN
INN is the GNU nntp server. It's a standalone application, that is it do not need an http server like Apache.
Also for french: doc en français
Searching
When searching for documentation, the "eyrie.org" is without a doubt the better documentation site about INN. so searching with:
word-to-search site:eyrie.org
like:
nocem site:eyrie.org
on google is a very effective way to find what you need.
Installation
Postfix
At some moment you may need a MTA, for example postfix
error <jdd@dodin.xyz> (expanded from <news>): SMTPUTF8 is required, but was not
offered by host dodin.xyz[109.234.161.13]
Postfix page about this error In summary:
- postconf smtputf8_enable=no
- postfix reload
Seams to fix the error
INN
openSUSE allows simple installation. Go to Yast, software, search for "InternetNews" and tick inn then OK... You get the basic setup. Leap 15.3 version (dec 2021) is 2.6.2-bp153.3.3.1.
INN is installed as user "news". In /etc/password:
news:[...]:News system:/etc/news:/sbin/nologin
So it's an user, but nobody can log in this account, and the account is in /etc. At some time in the INN use, you may have to work as "news" user. It's not as easy as one could think, but there is a workaroud: if you are root on your server, you can type
sudo -u news -s /bin/bash
and like this get a bash session as user news.
It's sometime difficult to find where the files are stored, every distribution have it's own places.
For openSUSE, files are in:
/etc/news (aka ~news, home of user "news"); you get in with "cd". /usr/lib/news (specially bin, where are tools and doc); /usr/share/doc/packages/inn; /var/log/news; /var/spool/news;
It may be useful to add in the ~news folder these links:
ln -s /var/spool/news spool ln -s /var/log/news logs ln -s /usr/lib/news/bin bin
To find them more easily and make command prefixes simpler.
To receive logs with mails, you have to install "mail" (or mailx), that means postfix.
zypper in mail (as root)
Configuration
A good followup is checklist summary below.
Firewall (as root)
First thing to do is open the firewall 119 port and add a route from the gateway if any.
May be also open port 25 for smtp if you want to share mails.
dont forget ssh :-).
Readers.conf
Then you have to make your groups available from the universe. So, go as user news, then add the following lines to the file readers.conf. It may be a good idea to paste this before the "abusers" line if ever you need to protect you from such bad guy.
auth "remote" {
hosts: "*"
default: "<remote>"
}
access "remote" {
users: "<remote>"
newsgroups: "*"
access: RP
}
Inn.conf
Change also inn.conf to add your name. Add also "localhost" or any domain name that fits to the line "server:" at first I was thinking it was by default, but it seems not in fact. You can also add "abuse" to the "complaints:" line just in case it's not default.
enableoverview: true #default
as default if the server have to be accessed by users and.
ovmethod tradindexed # default
You may add your domain name in the inn.conf file ("domain:"), but I'm not sure if it's mandatory.
/etc/aliases
On my openSUSE, the /etc/aliases file link abuse to postmaster, postmaster to root. So at some time you may have to link root to some email you are sure to read frequently.
Storage.conf
In storage.conf openSUSE default seems to be:
## By default, store everything in tradspool.
method tradspool {
newsgroups: *
class: 1
}
Create your first group
For example, as user news, type:
/usr/lib/news/bin/ctlinnd newgroup local.test
Check internally
You have to start INN for these commands to work.
To check the install, do "/usr/lib/news/bin/ctlinnd mode", "/usr/lib/news/bin/ctlinnd checkfile" and others. If you are asked to give "reason", type some words to identify the change.
inncheck -a
should give you hints on what don't works, having nothing is good.
inncheck -a -v -f --pedantic --perm
should give you hints about files permissions that have to be set, and in case of error give you the command that have to be used (as root).
Test from outside
with this you server may be visible from the net. But to be fair, you have to open the port 119 in your Firewall if any.
You can check this with:
telnet <your server> 119
it should respond
$ telnet dodin.xx 119 Trying 142.4.aaa.bbb... Connected to dodin.xx. Escape character is '^]'. 200 ns507557.dodin.xx InterNetNews NNRP server INN 2.6.2 ready (posting ok)
Then, list groups
LIST 215 Newsgroups in form "group high low status" control 0000000000 0000000001 n control.cancel 0000000000 0000000001 n control.checkgroups 0000000000 0000000001 n control.newgroup 0000000000 0000000001 n control.rmgroup 0000000000 0000000001 n junk 0000000000 0000000001 n test 0000000001 0000000001 y linux-31 0000000000 0000000001 y .
Then try to post a message (in red, it's the inn answer, type only what is in black):
POST
340 Ok, recommended message-ID <sosl56$13k$7@ns507557.dodin.xx>
From: jdd@dodin.org Subject: test Newsgroups: test (type a carriage return, then some text with CR between lines) test Fin . (this is the main keyboard dot, not the numerical pad one)
240 Article received <sosl56$13k$7@ns507557.dodin.xx>
Warning: Telnet is very picky, it don't allow nearly any fix to typing errors, The three headers are mandatory and have ton get the semi colon (:). If any error, type "quit" et do it again...
The final dot (.) send the message...
Get a look at what happen
One needs to check if the system works. INN is setup to send a daily summary to "news". It's feasible to resend this to root on your server and to send root mail to some external mail you happen to use to such task, with an alias in /etc/aliases
root: \root, myname@myserver.xxx
Debug
Articles
On my config, articles are stored in spool. spool is only a link to /var/spool/news.
Here you will find a file with the groups list (tradspool.map) and various places where articles are stored, for example "articles", where articles are simply numbered
> ll ~news/spool/articles/ total 0 drwxrwxr-x 1 news news 12 24 déc. 19:46 control drwxrwxr-x 1 news news 16 2 janv. 11:52 dodin drwxrwxr-x 1 news news 156 2 janv. 07:30 fr
Logs
My system uses systemd, but have also rsyslog installed.
The configuration file is /etc/rsyslog.conf and have the required news lines commented. As root, comment them out, then:
# systemctl restart rsyslog.service
Common logs are in /var/log/news, relevant lines are:
> ll ~news/logs/ -rw-r--r-- 1 news news 0 2 janv. 03:00 news.crit -rw-r--r-- 1 news news 0 2 janv. 03:00 news.err -rw-r--r-- 1 news news 0 2 janv. 03:00 news.notice
Mail gateway (Mailman)
The goal is to have a mailing-list mirrored on a newsgroup.
Posts have to be sent from the list to the group and from the group to the list as needed.
From list to news (Mailman)
(untested in 2022)
Pretty simple, in Mailman Web interface:
Mail gateway (Sympa)
Receiving the list
For all examples, we use a mailing list "test@culte.org" and a NNTP server "dodin.fr.nf" "dodin.fr.nf" MUST accept email (obviously), postfix (or equivalent) have to be configured and smtp (port 25) open in the Firewall if any.
From list to newsgroup
Group side as root
For a newsgroup, eg "test". Use the mailpost program (shipped with inn) to established the gateway. Create an alias in the MTA of dodin.fr.nf :
test: "|/usr/lib/news/bin/mailpost -b /var/spool/mailpost -t /var/tmp -d local culte.test"
then newaliases
debugging
# ll /var/spool/mailpost/
To see if mailpost writes there
# ll /var/spool/mailpost/ total 20 -rw-rw-r-- 1 nobody nobody 16 5 févr. 12:22 mailpost-msgid.dir -rw-rw-r-- 1 nobody nobody 16384 5 févr. 16:10 mailpost-msgid.pag
mailpost-msgid.pag changes (date, taille), not mailpost-msgid.dir
then send messages from root to test, then test@dodin.fr.nf. If they go through, good. Always add the post date in the text or the subject for reference
Then try from the list server. The "Sympa" one allows admin to send post directly from the admin web interface.
Try:
cat /var/log/mail* | grep "whatever"
Check that aliases works (see postfix configure here).
When a mail is sent to the list, test@dodin.fr.nf will receive it. And with mailpost, it will be posted on the newsgroup.
News side
As "news", create the group:
/usr/lib/news/bin/ctlinnd newgroup culte.test
debugging
execute often
inncheck -a -v -f --pedantic --perm
List side
Add a new subscriber to the mailing list. Its mail address must be handled by dodin.fr.nf. You can use "test@dodin.fr.nf".
From group to mailing list
sympa method (can't make it work: moderation is refused (not allowed to post))
It's quite simple : mark the group as moderated. This is done while the server is running with: ctlinnd changegroup test m For group "test" and letter m, for "moderated". On my server the group is "culte.test" to separate the groups of my Linux User Group (http://culte.org) from more general ones. And set the moderation address to the mailing list address (mylist@mylistserver.org). In the file "~news/moderators" add culte.test:test@culte.org Change the mailpost alias using the -a flag of mailpost (the -a flag marks the message as accepted by a moderator): test: "| /path/to/mailpost -a culte.test" Then, when a message is sent to the list, it is approved by the moderator and then posted. When a message is directly posted, it's sent to the moderator (which is the whole mailing list). As other messages on the list, it's approved and posted. This last method can be usefull to create a complete message system : mail, news and web interfaces can be used to read and send messages.
Mailman method (should work also for Sympa)
As "news" edit "newsfeeds"
The line following is commented out by default, uncomment it.
news2mail!:!*:Ac,Tc,Wn*:<pathbin>/news2mail
Is to take as it is, only "<pathbin>" is for us /usr/lib/news/bin/
So:
news2mail!:!*:Ac,Tc,Wn*:/usr/lib/news/bin/news2mail
then, for a "test" list and culte.test group:
test/culte.org:!*,culte.test:Ap,Tm:news2mail!
The first field (test) is only a random name, it only have to be the same as in the other file. "culte.org" is the list server domain name., Warning: no blank space at the end of the line!
there is no need to have the list name here, so you have to read "test/culte.org", not "test@culte.org"...
As "news" edit news2mail.cf
test test@culte.org test@dodin.fr.nf
First field (test) is the same than in the other file. The second field is the target list and the last is the "sender" name for the header. If the list is only for subscribers, use the mail of a legitimate list user.
Debugging
Syntax is important and not always easy to understand. You can use very often "inncheck -a", and even
inncheck -a -v -f --pedantic --perm
That should see even excess spaces at the end of a line
Cleaning and reporting - cronjobs
From "news"
It's necessary to setup cronjobs to clean articles periodically and some other housekeeping tasks.
On the news account, do:
crontab -e
Copy/paste:
0 3 * * * /usr/lib/news/bin/news.daily expireover delayrm lowmark 0,10,20,30,40,50 * * * * /usr/lib/news/bin/nntpsend 6 6 * * * /usr/lib/news/bin/ctlinnd flush inpaths! 10 6 * * * /usr/lib/news/bin/sendinpaths -c
Public sharing
To add public sharing of processed nocem,
- create a group, for example "dodin.processed"
Add a line in crontab
MAILTO = processed 50 23 * * * grep -B1 "processed notice" /etc/news/logs/news.notice
On openSUSE, logs are managed by rsyslog, it compresses the news.notice file at 23:55 every day. So why I need to run my grep a bit before.
From root
- add an alias in /et/aliases
processed: "|/usr/lib/news/bin/mailpost -b /var/tmp -t /var/tmp dodin.processed"
and run "newaliases".
Probably necessary, create a "processed" user on the server (without login), the one to receive the mail.
Start INN and make it permanent
To start inn:
systemctl start inn.service
To make INN permanent, one have to enable the service, in YaST or with:
systemctl enable inn.service
Filling the groups (peering)
Why and where
Usenet is a collaborative system: articles can be published anywhere and have to be spread al as many servers as it can, to reach users.
For my own config Peering with me (dodin.fr.nf, 2021)
Those servers are named "peers" and exchanging articles is "feeding".
Of course no server is obliged to feed yours, you have to ask permissions.
For the "Free.fr" french provider, the page http://nntpfeed.proxad.net/setup_fr.html
Retention time
On the past, bandwitdth and storage capacity where measured, it's no more really the case now (2022), with gigabytes storage and multicore processors.
So inf the only reason to remove old articles is space, one can keep them forever.
Using now only french groups (around 350 ones) makes it even more manageable.
So... don't change In inn.conf, setup artcutoff: 10 and in expire.ctl /remember/:11
If I understand well, these numbers are the time the ID of articles that are already removed for some reason are still kept, nothing we aim to change.
but edit the line
*:A:1:365:never <pattern>:<flag>:<min>:<default>:<max>
changing the 365 for any day number you want. This line means (see man page for details):
- pattern: * = all groups. You can have a rule for any group writing a similar line following this one.
- flag: select moderated or not, A is for all groups.
For the three other parts, the relevant text of the man page is:
"If <min> is set to "never", no article matching that line will ever be expired. If <default> is set to "never", no article matching that line without an explicit Expires: header will ever be expired. If <max> is set to "never", Expires: headers will be honored no matter how far into the future they are."
So no need to change first and last flag (1 and never), because is somebody set an expiry date in the headers, there may be a valuable reason and a clock may be out of date, the middle is for most article and can be a number of days or never (never expire).
Creating group list
It's done with actsync like this
actsync -o x -v 2 -p 0 -i /etc/news/actsync.ign dodin.fr.nf <peer server>
Example
For example, a well know server is aioe, and instructions are here.
The instructions shows changes to be done in three files, specially "newsfeed". Don't forget to uncomment the innfeed lines, like this (it was commented out in my file):
# innfeed funnel master.
innfeed!\
:!*\
:Tc,Wnm*:/usr/lib/news/bin/innfeed
It's also probably best to have in inn.conf:
pathhost: yourdomain.name
Sending statistics
It's well advised to send statistics about your server to the The Official TOP1000 Usenet Servers page.
For this to work you have to use ninpath
With my linux distribution (openSUSE Leap 15.3 on january 5, 2022), most of the work is already done, but you have to create the folder "path" yourself now, ie
mkdir ~news/logs/path/
An entry like this have to be in newsfeeds, but it may be already. If not, add:
inpaths!:*:Tc,WP:<pathbin>/ninpaths -p -d <pathlog>/path/inpaths.%d"
To start the system, type as news (the 'in...' is just a reason you can change):
ctlinnd reload newsfeeds 'inpaths feed setup'
and you should see a new file in ~news/logs/path/.
Then add the two lines in crontab (crontab -e):
6 6 * * * /usr/lib/news/bin/ctlinnd flush inpaths! 10 6 * * * /usr/lib/news/bin/sendinpaths -c
You can test if if works with
sendinpaths -c
This should send the statistics to the top1000 server and to your newsmaster mail, and possibly remove the file(s) in ~news/logs/path/
Control: groups modifications
This part is for french usenet.
Control articles are pretty rare: they are used for creation or deletion of groups, worldwide. French ones are managed by a french team and a people nicknamed "Control" (surprise :-!). Control is the only one allowed to publish such control message.
To be sure control articles are really coming from him, the server have to import a valid pgp key doing this as "news":
wget http://www.usenet-fr.net/pgp-fr-2020.txt gpg --import pgp-fr-2020.txt
then you can test to see if it worked by:
gpg --list-keys
you have to get this result:
pub rsa4096 2020-11-20 [SC]
D5F369B297573622015354E7FA423E89970EB10B
uid [ inconnue] control@usenet-fr.news.eu.org
sub rsa4096 2020-11-20 [E]
And the files in ~news/.gnupg have to be dated today.
If you have a recent enough INN install the relevant config file options are already active. Else check the link above.
Filtering spam
Cleanfeed
aioe recommends the "cleanfeed" spam filter. This one is not available on my openSUSE Leap 15.3, but it's a simple perl script one can find here.
The filter folder as said in inn.conf pathfilter option is "/usr/lib/news/bin/filter", for convenience, I added in my ~news folder a link and changed the ownership to news:news.
ln -s /usr/lib/news/bin/filter filter
then I noticed that the "filter" folder is not empty... but the /usr/lib/news/bin/filter folder is owned by root, so dealing with it needs to be root.
Downloading cleanfeed
So go to the link above.
- download the zip file from "code" with "wget -nH https://github.com/crooks/cleanfeed/archive/refs/heads/master.zip", this create the master.zip file in your present folder (notice there are several versions of cleanfeed archive, mostly indentical);
- use mc to open this file and copy "cleanfeed-master" to the news folder. In this way all the relevant files/folders are created.
The download can give you a "cleanfeed.tar.gz", a master.zip or a "cleanfeed.zip", but the content is the same.
news@ns507557:~> ll cleanfeed/ total 120 drwxr-xr-x 1 news news 84 2 mars 2020 badurls -rw-r--r-- 1 news news 1577 2 mars 2020 changelog -rw-r--r-- 1 news news 96948 2 mars 2020 cleanfeed -rw-r--r-- 1 news news 197 2 mars 2020 cleanfeed.asc -rw-r--r-- 1 news news 7035 2 mars 2020 cleanfeed.local.sample -rw-r--r-- 1 news news 749 2 mars 2020 filter_nnrpd.pl drwxr-xr-x 1 news news 234 2 mars 2020 html -rw-r--r-- 1 news news 838 2 mars 2020 README drwxr-xr-x 1 news news 216 2 mars 2020 samples drwxr-xr-x 1 news news 28 2 mars 2020 tools
- go as root to /usr/lib/news/bin/filter;
- moved the present content to an "oldfilter" folder (just in case);
- ln -s /etc/news/cleanfeed-master/filter_nnrpd.pl filter_innd.pl
s507557:/usr/lib/news/bin/filter # ll total 4 lrwxrwxrwx 1 root root 42 20 déc. 10:22 filter_nnrpd.pl -> /etc/news/cleanfeed-master/filter_nnrpd.pl drwxr-xr-x 1 root news 288 19 déc. 18:32 oldfilter
< Postfix configure (2021 edition) | Configuring a small home server in 2021 with openSUSE Leap 15.3 | MySQL - Mariadb 2021 >
Allowing perl and filter_innd.pl
The doc is here and here for cleanfeed.
The culprit is than
- perl can't be enabled in INN if there is no perl filter... the default install gives the necessary files, so thatb perl is enabled, but as soon as you move the files to an "oldfiles" folder, perl is disabled until filter_innd.pl is again created.
- cleanfeed filter is *not" filter_innd.pl but simply "cleanfeed" in the cleanfeed folder. This may seems obvious, but it's not as there *is* a filter_nnrpd.pl file.
So
news@ns507557:/etc/news/filter> ln -s /etc/news/cleanfeed-master/cleanfeed filter_innd.pl news@ns507557:/etc/news/filter> ll total 8 lrwxrwxrwx 1 news news 36 24 déc. 08:52 filter_innd.pl -> /etc/news/cleanfeed-master/cleanfeed lrwxrwxrwx 1 news news 42 22 déc. 09:53 filter_nnrpd.pl -> /etc/news/cleanfeed-master/filter_nnrpd.pl drwxr-xr-x 1 news news 288 19 déc. 18:32 oldfilter news@ns507557:/etc/news/filter> ctlinnd reload filter.perl 'reason' Ok news@ns507557:/etc/news/filter> ctlinnd perl y Ok news@ns507557:/etc/news/filter> ctlinnd mode Server running Allowing remote connections Parameters c 10 i 50 (1) l 5000000 o 1010 t 300 H 2 T 60 X 0 normal specified Not reserved Readers follow enabled Perl filtering enabled Perl filter stats: Pass: 0 Reject: 0 Refuse: 0 MD5: 0 PHL: 0 PHN: 0 PHR: 0 FSL: 0
Fine tuning security (authentication, nocem, cancel)
Authentication
Authentication is a way of knowing who can post to your server. AFAIK, INN can use most usual systems. On Linux, the simpler is to create a system account for each user, but you may not want to do this.
In fact I would like to keep my server free access - but it's unlikely I can.
Nocem
Peers
"Nocem" is a new (?) system to manage usenet articles. here the project home page (may be obsolete, the alt.nocem.misc group is empty.
To use it you have to have to find people that make nocem article. To be very short, nocem articles defines some sort of filter allowing to find and remove unwanted articles.
The nocem article sent to me are in the dodin.nocem local group. It's likely that the server you use have same kind of local group, and the references can be found inside the nocem messages.
Nocem and pgp
Nocem uses articles authentified with pgp, so pgp have to work. Looks like the Debian INN package do this, but not the openSUSE packege.
pgp is necessary to have the perl-nocem script work.
So you may have to edit "/usr/lib/news/lib/perl/INN/Config.pm" (<pathlib>/perl/INN/Config.pm), adding /usr/bin/xxx to get
our $gpgv = '/usr/bin/gpgv'; our $gpg = '/usr/bin/gpg'; our $pgp = '/usr/bin/pgp';
Nocem group
1) Create a nocem group:
ctlinnd newgroup dodin.nocem
Import the pgp key
What I did:
- create the folder ~news/pgp
- cd ~news/pgp
- create the ncmring file with :touch ncmring.gpg
- wget http://llpminibo<link to the pgp public key>
Edit newsfeeds
nocem!:!*,dodin.nocem:Tc,Wf,Ap:/usr/lib/news/bin/perl-nocem
Edit ~news/nocem.ctl
Add the line:
llpmixxx@free.fr:cleaning
That is the nocem author mail, ":", the list of nocem articles accepted. The word (here "cleaning") have to be found from the message author.
Restart
At least INN, may be the hole server if pgp or perl where just installed.
look at the logs
news> grep -R nocem * | less
may give you a long list of places where "nocem" is quoted, including messages.
Cancel-lock, cancel-key
Basically, anybody having the necessary knowledge can issue a cancel to remove an article from a group. To prevent such thing, a system called "cancel-lock, cancel-key" was created and will be default on INN 2.7 (I'm still in 2.6 on feb 20, 2022).
Right now it need to use a patch of cleanfeed, you can find here.
For me it was in "cleanfeed-master", I used (but check the link to see if it's still the exact link)
wget https://home.gegeweb.org/files/filter_nnrpd.pl
and
wget https://home.gegeweb.org/files/cleanfeed.local
Make sure you have the Digest::SHA and Digest::MD5 Perl modules installed. They are normally standard modules but some distributions do not install it by default.
The following commands should not return any error: % perl -e "use Digest::SHA" % perl -e "use Digest::MD5"
What are "cancels"
...and how forgive them
usenet format on templetons.com
In inn.conf:
"In order not to actually process any cancel or supersedes messages, you can start innd with the -C flag, or add this flag to the innflags parameter. "
so:
innflags: "-C"