Doc /

Configure the INN nntp server (2021/2022)

< Postfix configure (2021 edition) | Configuring a small home server in 2021 with openSUSE Leap 15.3 | MySQL - Mariadb 2021 >

Références

RFC

Usenet Article Format

INN

INN is the GNU nntp server. It's a standalone application, that is it do not need an http server like Apache.

Also for french: doc en français

Searching

When searching for documentation, the "eyrie.org" is without a doubt the better documentation site about INN. so searching with:

word-to-search site:eyrie.org

like:

nocem site:eyrie.org

on google is a very effective way to find what you need.

Installation

Postfix

At some moment you may need a MTA, for example postfix

error <jdd@dodin.xyz> (expanded from <news>): SMTPUTF8 is required, but was not
    offered by host dodin.xyz[109.234.161.13]

Postfix page about this error In summary:

  1. postconf smtputf8_enable=no
  2. postfix reload

Seams to fix the error

INN

openSUSE allows simple installation. Go to Yast, software, search for "InternetNews" and tick inn then OK... You get the basic setup. Leap 15.3 version (dec 2021) is 2.6.2-bp153.3.3.1.

INN is installed as user "news". In /etc/password:

 news:[...]:News system:/etc/news:/sbin/nologin

So it's an user, but nobody can log in this account, and the account is in /etc. At some time in the INN use, you may have to work as "news" user. It's not as easy as one could think, but there is a workaroud: if you are root on your server, you can type

sudo -u news -s /bin/bash

and like this get a bash session as user news.

It's sometime difficult to find where the files are stored, every distribution have it's own places.

For openSUSE, files are in:

/etc/news (aka ~news, home of user "news"); you get in with "cd".
/usr/lib/news (specially bin, where are tools and doc);
/usr/share/doc/packages/inn;
/var/log/news;
/var/spool/news;

It may be useful to add in the ~news folder these links:

ln -s /var/spool/news spool
ln -s /var/log/news logs
ln -s /usr/lib/news/bin bin

To find them more easily and make command prefixes simpler.

mail

To receive logs with mails, you have to install "mail" (or mailx), that means postfix.

zypper in mail (as root)

Configuration

A good followup is checklist summary below.

Firewall (as root)

First thing to do is open the firewall 119 port and add a route from the gateway if any.

May be also open port 25 for smtp if you want to share mails.

dont forget ssh :-).

Readers.conf

Then you have to make your groups available from the universe. So, go as user news, then add the following lines to the file readers.conf. It may be a good idea to paste this before the "abusers" line if ever you need to protect you from such bad guy.

auth "remote" {
    hosts: "*"
    default: "<remote>"
}

access "remote" {
    users: "<remote>"
    newsgroups: "*"
    access: RP
}

Inn.conf

Change also inn.conf to add your name. Add also "localhost" or any domain name that fits to the line "server:" at first I was thinking it was by default, but it seems not in fact. You can also add "abuse" to the "complaints:" line just in case it's not default.

enableoverview: true #default

as default if the server have to be accessed by users and.

ovmethod tradindexed # default

You may add your domain name in the inn.conf file ("domain:"), but I'm not sure if it's mandatory.

/etc/aliases

On my openSUSE, the /etc/aliases file link abuse to postmaster, postmaster to root. So at some time you may have to link root to some email you are sure to read frequently.

Storage.conf

In storage.conf openSUSE default seems to be:

##  By default, store everything in tradspool.

method tradspool {
    newsgroups: *
    class: 1
}

Create your first group

For example, as user news, type:

 /usr/lib/news/bin/ctlinnd newgroup local.test

Check internally

You have to start INN for these commands to work.

To check the install, do "/usr/lib/news/bin/ctlinnd mode", "/usr/lib/news/bin/ctlinnd checkfile" and others. If you are asked to give "reason", type some words to identify the change.

 inncheck -a

should give you hints on what don't works, having nothing is good.

 inncheck -a -v -f --pedantic --perm

should give you hints about files permissions that have to be set, and in case of error give you the command that have to be used (as root).

Test from outside

with this you server may be visible from the net. But to be fair, you have to open the port 119 in your Firewall if any.

You can check this with:

 telnet <your server> 119

it should respond

$ telnet dodin.xx 119
Trying 142.4.aaa.bbb...
Connected to dodin.xx.
Escape character is '^]'.
200 ns507557.dodin.xx InterNetNews NNRP server INN 2.6.2 ready (posting ok)

Then, list groups

LIST      
215 Newsgroups in form "group high low status"
control 0000000000 0000000001 n
control.cancel 0000000000 0000000001 n
control.checkgroups 0000000000 0000000001 n
control.newgroup 0000000000 0000000001 n
control.rmgroup 0000000000 0000000001 n
junk 0000000000 0000000001 n
test 0000000001 0000000001 y
linux-31 0000000000 0000000001 y
.

Then try to post a message (in red, it's the inn answer, type only what is in black):

 POST

340 Ok, recommended message-ID <sosl56$13k$7@ns507557.dodin.xx>

 From: jdd@dodin.org
 Subject: test
 Newsgroups: test
 (type a carriage return, then some text with CR between lines)
 test
 Fin
 . (this is the main keyboard dot, not the numerical pad one)

240 Article received <sosl56$13k$7@ns507557.dodin.xx>

Warning: Telnet is very picky, it don't allow nearly any fix to typing errors, The three headers are mandatory and have ton get the semi colon (:). If any error, type "quit" et do it again...

The final dot (.) send the message...

Get a look at what happen

One needs to check if the system works. INN is setup to send a daily summary to "news". It's feasible to resend this to root on your server and to send root mail to some external mail you happen to use to such task, with an alias in /etc/aliases

 root: \root, myname@myserver.xxx

Debug

Articles

On my config, articles are stored in spool. spool is only a link to /var/spool/news.

Here you will find a file with the groups list (tradspool.map) and various places where articles are stored, for example "articles", where articles are simply numbered

> ll ~news/spool/articles/
total 0
drwxrwxr-x 1 news news  12 24 déc.  19:46 control
drwxrwxr-x 1 news news  16  2 janv. 11:52 dodin
drwxrwxr-x 1 news news 156  2 janv. 07:30 fr

Logs

My system uses systemd, but have also rsyslog installed.

The configuration file is /etc/rsyslog.conf and have the required news lines commented. As root, comment them out, then:

 # systemctl restart rsyslog.service 

Common logs are in /var/log/news, relevant lines are:

> ll ~news/logs/

-rw-r--r-- 1 news news     0  2 janv. 03:00 news.crit
-rw-r--r-- 1 news news     0  2 janv. 03:00 news.err
-rw-r--r-- 1 news news     0  2 janv. 03:00 news.notice

Mail gateway (Mailman)

The goal is to have a mailing-list mirrored on a newsgroup.

Posts have to be sent from the list to the group and from the group to the list as needed.

From list to news (Mailman)

(untested in 2022)

Pretty simple, in Mailman Web interface:

My own Mailman doc

Mail gateway (Sympa)

Sympa comments

Receiving the list

For all examples, we use a mailing list "test@culte.org" and a NNTP server "dodin.fr.nf" "dodin.fr.nf" MUST accept email (obviously), postfix (or equivalent) have to be configured and smtp (port 25) open in the Firewall if any.

From list to newsgroup

Group side as root

For a newsgroup, eg "test". Use the mailpost program (shipped with inn) to established the gateway. Create an alias in the MTA of dodin.fr.nf :

test: "|/usr/lib/news/bin/mailpost -b /var/spool/mailpost -t /var/tmp -d local culte.test"

then newaliases

debugging

 # ll /var/spool/mailpost/

To see if mailpost writes there

# ll /var/spool/mailpost/
total 20
-rw-rw-r-- 1 nobody nobody    16  5 févr. 12:22 mailpost-msgid.dir
-rw-rw-r-- 1 nobody nobody 16384  5 févr. 16:10 mailpost-msgid.pag

mailpost-msgid.pag changes (date, taille), not mailpost-msgid.dir

then send messages from root to test, then test@dodin.fr.nf. If they go through, good. Always add the post date in the text or the subject for reference

Then try from the list server. The "Sympa" one allows admin to send post directly from the admin web interface.

Try:

cat /var/log/mail* | grep "whatever"

Check that aliases works (see postfix configure here).

When a mail is sent to the list, test@dodin.fr.nf will receive it. And with mailpost, it will be posted on the newsgroup.

News side

As "news", create the group:

/usr/lib/news/bin/ctlinnd newgroup culte.test

debugging

execute often

inncheck -a -v -f --pedantic --perm

List side

Add a new subscriber to the mailing list. Its mail address must be handled by dodin.fr.nf. You can use "test@dodin.fr.nf".

From group to mailing list

sympa method (can't make it work: moderation is refused (not allowed to post))

It's quite simple : mark the group as moderated. This is done while the server is running with:
ctlinnd changegroup test m
For group "test" and letter m, for "moderated". On my server the group is "culte.test" to separate the groups of my Linux User Group (http://culte.org) from more general ones.
And set the moderation address to the mailing list address (mylist@mylistserver.org).
In the file "~news/moderators" add
culte.test:test@culte.org
Change the mailpost alias using the -a flag of mailpost (the -a flag marks the message as accepted by a moderator):
test: "| /path/to/mailpost -a culte.test"
Then, when a message is sent to the list, it is approved by the moderator and then posted. When a message is directly posted, it's sent to the moderator (which is the whole mailing list). As other messages on the list, it's approved and posted.
This last method can be usefull to create a complete message system : mail, news and web interfaces can be used to read and send messages.

Mailman method (should work also for Sympa)

Most of the doc is here:

As "news" edit "newsfeeds"

The line following is commented out by default, uncomment it.

news2mail!:!*:Ac,Tc,Wn*:<pathbin>/news2mail

Is to take as it is, only "<pathbin>" is for us /usr/lib/news/bin/

So:

news2mail!:!*:Ac,Tc,Wn*:/usr/lib/news/bin/news2mail

then, for a "test" list and culte.test group:

test/culte.org:!*,culte.test:Ap,Tm:news2mail!

The first field (test) is only a random name, it only have to be the same as in the other file. "culte.org" is the list server domain name., Warning: no blank space at the end of the line!

there is no need to have the list name here, so you have to read "test/culte.org", not "test@culte.org"...

As "news" edit news2mail.cf

test test@culte.org test@dodin.fr.nf

First field (test) is the same than in the other file. The second field is the target list and the last is the "sender" name for the header. If the list is only for subscribers, use the mail of a legitimate list user.

Debugging

Syntax is important and not always easy to understand. You can use very often "inncheck -a", and even

inncheck -a -v -f --pedantic --perm

That should see even excess spaces at the end of a line

Cleaning and reporting - cronjobs

From "news"

It's necessary to setup cronjobs to clean articles periodically and some other housekeeping tasks.

On the news account, do:

 crontab -e

Copy/paste:

0 3 * * * /usr/lib/news/bin/news.daily expireover delayrm lowmark
0,10,20,30,40,50 * * * * /usr/lib/news/bin/nntpsend
6   6 * * *   /usr/lib/news/bin/ctlinnd flush inpaths!
10  6 * * *   /usr/lib/news/bin/sendinpaths -c

Public sharing

To add public sharing of processed nocem,

  • create a group, for example "dodin.processed"

Add a line in crontab

MAILTO = processed
50 23 * * * grep -B1 "processed notice" /etc/news/logs/news.notice

On openSUSE, logs are managed by rsyslog, it compresses the news.notice file at 23:55 every day. So why I need to run my grep a bit before.

From root

  • add an alias in /et/aliases
processed: "|/usr/lib/news/bin/mailpost -b /var/tmp -t /var/tmp dodin.processed"

and run "newaliases".

Probably necessary, create a "processed" user on the server (without login), the one to receive the mail.

Start INN and make it permanent

To start inn:

 systemctl start inn.service

To make INN permanent, one have to enable the service, in YaST or with:

 systemctl enable inn.service

Filling the groups (peering)

Why and where

Usenet is a collaborative system: articles can be published anywhere and have to be spread al as many servers as it can, to reach users.

For my own config Peering with me (dodin.fr.nf, 2021)

Those servers are named "peers" and exchanging articles is "feeding".

Of course no server is obliged to feed yours, you have to ask permissions.

For the "Free.fr" french provider, the page http://nntpfeed.proxad.net/setup_fr.html

Retention time

On the past, bandwitdth and storage capacity where measured, it's no more really the case now (2022), with gigabytes storage and multicore processors.

So inf the only reason to remove old articles is space, one can keep them forever.

Using now only french groups (around 350 ones) makes it even more manageable.

So... don't change In inn.conf, setup artcutoff: 10 and in expire.ctl /remember/:11

If I understand well, these numbers are the time the ID of articles that are already removed for some reason are still kept, nothing we aim to change.

but edit the line

 *:A:1:365:never

 <pattern>:<flag>:<min>:<default>:<max>

changing the 365 for any day number you want. This line means (see man page for details):

  • pattern: * = all groups. You can have a rule for any group writing a similar line following this one.
  • flag: select moderated or not, A is for all groups.

For the three other parts, the relevant text of the man page is:

"If <min> is set to "never", no article matching that line will ever be expired. If <default> is set to "never", no article matching that line without an explicit Expires: header will ever be expired. If <max> is set to "never", Expires: headers will be honored no matter how far into the future they are."

So no need to change first and last flag (1 and never), because is somebody set an expiry date in the headers, there may be a valuable reason and a clock may be out of date, the middle is for most article and can be a number of days or never (never expire).

Creating group list

It's done with actsync like this

 actsync -o x -v 2 -p 0 -i /etc/news/actsync.ign dodin.fr.nf <peer server>

Example

For example, a well know server is aioe, and instructions are here.

The instructions shows changes to be done in three files, specially "newsfeed". Don't forget to uncomment the innfeed lines, like this (it was commented out in my file):

# innfeed funnel master.
innfeed!\
        :!*\
        :Tc,Wnm*:/usr/lib/news/bin/innfeed

It's also probably best to have in inn.conf:

 pathhost: yourdomain.name

Sending statistics

It's well advised to send statistics about your server to the The Official TOP1000 Usenet Servers page.

For this to work you have to use ninpath

With my linux distribution (openSUSE Leap 15.3 on january 5, 2022), most of the work is already done, but you have to create the folder "path" yourself now, ie

 mkdir ~news/logs/path/

An entry like this have to be in newsfeeds, but it may be already. If not, add:

 inpaths!:*:Tc,WP:<pathbin>/ninpaths -p -d <pathlog>/path/inpaths.%d"

To start the system, type as news (the 'in...' is just a reason you can change):

 ctlinnd reload newsfeeds 'inpaths feed setup'

and you should see a new file in ~news/logs/path/.

Then add the two lines in crontab (crontab -e):

 6   6 * * *   /usr/lib/news/bin/ctlinnd flush inpaths!
 10  6 * * *   /usr/lib/news/bin/sendinpaths -c

You can test if if works with

 sendinpaths -c

This should send the statistics to the top1000 server and to your newsmaster mail, and possibly remove the file(s) in ~news/logs/path/

Control: groups modifications

This part is for french usenet.

Original article on usenet-fr

Control articles are pretty rare: they are used for creation or deletion of groups, worldwide. French ones are managed by a french team and a people nicknamed "Control" (surprise :-!). Control is the only one allowed to publish such control message.

To be sure control articles are really coming from him, the server have to import a valid pgp key doing this as "news":

wget http://www.usenet-fr.net/pgp-fr-2020.txt
gpg --import pgp-fr-2020.txt

then you can test to see if it worked by:

 gpg --list-keys

you have to get this result:

pub   rsa4096 2020-11-20 [SC]
      D5F369B297573622015354E7FA423E89970EB10B
uid          [ inconnue] control@usenet-fr.news.eu.org
sub   rsa4096 2020-11-20 [E]

And the files in ~news/.gnupg have to be dated today.

If you have a recent enough INN install the relevant config file options are already active. Else check the link above.

Filtering spam

Cleanfeed

aioe recommends the "cleanfeed" spam filter. This one is not available on my openSUSE Leap 15.3, but it's a simple perl script one can find here.

The filter folder as said in inn.conf pathfilter option is "/usr/lib/news/bin/filter", for convenience, I added in my ~news folder a link and changed the ownership to news:news.

 ln -s /usr/lib/news/bin/filter filter

then I noticed that the "filter" folder is not empty... but the /usr/lib/news/bin/filter folder is owned by root, so dealing with it needs to be root.

Downloading cleanfeed

So go to the link above.

  • download the zip file from "code" with "wget -nH https://github.com/crooks/cleanfeed/archive/refs/heads/master.zip", this create the master.zip file in your present folder (notice there are several versions of cleanfeed archive, mostly indentical);
  • use mc to open this file and copy "cleanfeed-master" to the news folder. In this way all the relevant files/folders are created.

The download can give you a "cleanfeed.tar.gz", a master.zip or a "cleanfeed.zip", but the content is the same.

news@ns507557:~> ll cleanfeed/
total 120
drwxr-xr-x 1 news news    84  2 mars   2020 badurls
-rw-r--r-- 1 news news  1577  2 mars   2020 changelog
-rw-r--r-- 1 news news 96948  2 mars   2020 cleanfeed
-rw-r--r-- 1 news news   197  2 mars   2020 cleanfeed.asc
-rw-r--r-- 1 news news  7035  2 mars   2020 cleanfeed.local.sample
-rw-r--r-- 1 news news   749  2 mars   2020 filter_nnrpd.pl
drwxr-xr-x 1 news news   234  2 mars   2020 html
-rw-r--r-- 1 news news   838  2 mars   2020 README
drwxr-xr-x 1 news news   216  2 mars   2020 samples
drwxr-xr-x 1 news news    28  2 mars   2020 tools
  • go as root to /usr/lib/news/bin/filter;
  • moved the present content to an "oldfilter" folder (just in case);
  • ln -s /etc/news/cleanfeed-master/filter_nnrpd.pl filter_innd.pl
s507557:/usr/lib/news/bin/filter # ll
total 4
lrwxrwxrwx 1 root root  42 20 déc.  10:22 filter_nnrpd.pl -> /etc/news/cleanfeed-master/filter_nnrpd.pl
drwxr-xr-x 1 root news 288 19 déc.  18:32 oldfilter

< Postfix configure (2021 edition) | Configuring a small home server in 2021 with openSUSE Leap 15.3 | MySQL - Mariadb 2021 >

Allowing perl and filter_innd.pl

The doc is here and here for cleanfeed.

The culprit is than

  • perl can't be enabled in INN if there is no perl filter... the default install gives the necessary files, so thatb perl is enabled, but as soon as you move the files to an "oldfiles" folder, perl is disabled until filter_innd.pl is again created.
  • cleanfeed filter is *not" filter_innd.pl but simply "cleanfeed" in the cleanfeed folder. This may seems obvious, but it's not as there *is* a filter_nnrpd.pl file.

So

news@ns507557:/etc/news/filter> ln -s /etc/news/cleanfeed-master/cleanfeed filter_innd.pl

news@ns507557:/etc/news/filter> ll
total 8
lrwxrwxrwx 1 news news  36 24 déc.  08:52 filter_innd.pl -> /etc/news/cleanfeed-master/cleanfeed
lrwxrwxrwx 1 news news  42 22 déc.  09:53 filter_nnrpd.pl -> /etc/news/cleanfeed-master/filter_nnrpd.pl
drwxr-xr-x 1 news news 288 19 déc.  18:32 oldfilter

news@ns507557:/etc/news/filter> ctlinnd reload filter.perl 'reason'
Ok

news@ns507557:/etc/news/filter> ctlinnd perl y
Ok

news@ns507557:/etc/news/filter> ctlinnd mode
Server running
Allowing remote connections
Parameters c 10 i 50 (1) l 5000000 o 1010 t 300 H 2 T 60 X 0 normal specified
Not reserved
Readers follow enabled
Perl filtering enabled
Perl filter stats: Pass: 0  Reject: 0  Refuse: 0  MD5: 0  PHL: 0  PHN: 0  PHR: 0  FSL: 0

Fine tuning security (authentication, nocem, cancel)

Authentication

Authentication is a way of knowing who can post to your server. AFAIK, INN can use most usual systems. On Linux, the simpler is to create a system account for each user, but you may not want to do this.

In fact I would like to keep my server free access - but it's unlikely I can.

Nocem

See the eyrie.org article

Peers

"Nocem" is a new (?) system to manage usenet articles. here the project home page (may be obsolete, the alt.nocem.misc group is empty.

To use it you have to have to find people that make nocem article. To be very short, nocem articles defines some sort of filter allowing to find and remove unwanted articles.

The nocem article sent to me are in the dodin.nocem local group. It's likely that the server you use have same kind of local group, and the references can be found inside the nocem messages.

Nocem and pgp

Nocem uses articles authentified with pgp, so pgp have to work. Looks like the Debian INN package do this, but not the openSUSE packege.

pgp is necessary to have the perl-nocem script work.

So you may have to edit "/usr/lib/news/lib/perl/INN/Config.pm" (<pathlib>/perl/INN/Config.pm), adding /usr/bin/xxx to get

our $gpgv = '/usr/bin/gpgv';
our $gpg = '/usr/bin/gpg';
our $pgp = '/usr/bin/pgp';

Nocem group

1) Create a nocem group:

ctlinnd newgroup dodin.nocem

Import the pgp key

What I did:

  • create the folder ~news/pgp
  • cd ~news/pgp
  • create the ncmring file with :touch ncmring.gpg
  • wget http://llpminibo<link to the pgp public key>

Edit newsfeeds

nocem!:!*,dodin.nocem:Tc,Wf,Ap:/usr/lib/news/bin/perl-nocem

Edit ~news/nocem.ctl

Add the line:

llpmixxx@free.fr:cleaning

That is the nocem author mail, ":", the list of nocem articles accepted. The word (here "cleaning") have to be found from the message author.

Restart

At least INN, may be the hole server if pgp or perl where just installed.

look at the logs

news> grep -R nocem * | less

may give you a long list of places where "nocem" is quoted, including messages.

Cancel-lock, cancel-key

Basically, anybody having the necessary knowledge can issue a cancel to remove an article from a group. To prevent such thing, a system called "cancel-lock, cancel-key" was created and will be default on INN 2.7 (I'm still in 2.6 on feb 20, 2022).

Right now it need to use a patch of cleanfeed, you can find here.

For me it was in "cleanfeed-master", I used (but check the link to see if it's still the exact link)

wget https://home.gegeweb.org/files/filter_nnrpd.pl

and

wget https://home.gegeweb.org/files/cleanfeed.local

The RFC is here

Make sure you have the Digest::SHA and Digest::MD5 Perl modules installed. They are normally standard modules but some distributions do not install it by default.

The following commands should not return any error:
% perl -e "use Digest::SHA"
% perl -e "use Digest::MD5"

What are "cancels"

...and how forgive them

faq cancels on "killfile.org"

usenet format on templetons.com

Howto cancel articles

In inn.conf:

"In order not to actually process any cancel or supersedes messages, you can start innd with the -C flag, or add this flag to the innflags parameter. "

so:

 innflags: "-C"