Discussion:
Need help with LXC container routing issues in Slackware 14.2
(too old to reply)
Lew Pitcher
2024-05-01 15:11:45 UTC
Permalink
Hi, gals and guys

As I've mentioned before, I'm late to the container party
and starting to play around with LXC containers. This time,
my goal is to model a container setup for my internet-facing
server, to provide better security for its systems. More
specifically, I am trying to model a layout which puts
both my internet connection, and various internet-facing
services into containers such that the arrangement provides
a "bastion host" fronting a set of services running in a
"DMZ" network, sandwitched between firewalls.

Graphically, the concept looks like:

Internet
|
=====(firewall)
Bastion Host
|
-----+----- ...
| |
=== === .....
me server


To this end, I've put together a couple of LXC containers,
one ("sysdev1") representing the "bastion host", and the
other ("sysdev2") representing a service within the "DMZ"

sysdev1 resides on two networks: my internal network (which
represents the big bad internet), and a "private" network
(implemented as veth devices talking through a bridge
on the hosting system).

sysdev2 resides on one network: the "private" network
that sysdev1 resides on.

I have proven that sysdev1 has a route to the "internet"
(actually the Internet, but for now we can ignore the
part outside of my network), so I've managed to prove
that
1) I have correctly configured ip_forward on my host
2) I have correctly implemented routing on my host
3) I have correctly implemented routing (outbound)
in the sysdev1 container, including setting the
default gateway.
I have also tested the route from sysdev1 to sysdev2,
so I can also say that
4) I have correctly implemented the "private network"
routing in the sysdev1 container.

From sysdev2, I can ping & traceroute to sysdev1, so
5) I have correctly implemented the "private network"
routing in the sysdev2 container.
But, I cannot ping or traceroute beyond that private
network.

In sysdev1, I have the "gateway" set up to point
to my host system's address, and ip_forward is
properly set (inherited from the host system).

In sysdev2, I have the "gateway" set up to point
to sysdev1, and (again) ip_forward is properly
set (again, inherited from the host system).

As far as I can tell, I've got all the proper
routing rules in place, but I still cannot
route from sysdev2 to any system past sysdev1.

I've attached the technical details (routing table,
interface list, tests) below.

So, to the question: What have I done wrong, or
not done yet? Any advice you can offer would be
appreciated.

Thanks. Lew


==== sysdev2 ON 192.168.55.2 ====
Note: sysdev2 is the "services" standin within the "DMZ"

Script started on Wed 01 May 2024 10:06:47 AM EDT
***@sysdev2:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.55.1 0.0.0.0 UG 1 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.55.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
***@sysdev2:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.2 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::d076:b7ff:fe55:8543 prefixlen 64 scopeid 0x20<link>
ether d2:76:b7:55:85:43 txqueuelen 1000 (Ethernet)
RX packets 25 bytes 2855 (2.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 35 bytes 3766 (3.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

***@sysdev2:~# cat /proc/sys/net/ipv4/ip_forward
1
***@sysdev2:~# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.55.1 0.086 ms 0.017 ms 0.014 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 *^C
***@sysdev2:~# exit
exit

Script done on Wed 01 May 2024 10:07:21 AM EDT


==== sysdev1 ON 192.168.55.2 / 192.168.99.32 ====
Note: sysdev1 is the "bastion host" standin that fronts the "DMZ"

Script started on Wed 01 May 2024 10:08:17 AM EDT
***@sysdev1:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.99.3 0.0.0.0 UG 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.55.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
***@sysdev1:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.99.32 netmask 255.255.255.0 broadcast 192.168.99.255
inet6 fe80::90da:85ff:fecf:24cc prefixlen 64 scopeid 0x20<link>
ether 92:da:85:cf:24:cc txqueuelen 1000 (Ethernet)
RX packets 266 bytes 28495 (27.8 KiB)
RX errors 0 dropped 1 overruns 0 frame 0
TX packets 277 bytes 28000 (27.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.1 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::c4a:71ff:fe6c:323a prefixlen 64 scopeid 0x20<link>
ether 0e:4a:71:6c:32:3a txqueuelen 1000 (Ethernet)
RX packets 88 bytes 8308 (8.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 43 bytes 3795 (3.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 37 bytes 3328 (3.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 37 bytes 3328 (3.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

***@sysdev1:~# cat /proc/sys/net/ipv4/ip_forward
1
***@sysdev1:~# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.99.3 0.061 ms 0.022 ms 0.018 ms
2 192.168.99.1 0.150 ms 0.127 ms 0.127 ms
3 my.isp.net.addr 12.614 ms 13.526 ms 16.187 ms
... elided for privacy ...
8 8.8.8.8 37.242 ms 26.502 ms 142.251.70.11 27.062 ms
***@sysdev1:~# exit
exit

Script done on Wed 01 May 2024 10:09:02 AM EDT


==== wordsworth ON 192.168.99.3 ====
Note: wordsworth is the "internet" standin that converses
with the "bastion host". It also is the host of the
"sysdev1" and "sysdev2" containers, and the bridge
that implements the DMZ "private network"

Script started on Wed 01 May 2024 10:10:55 AM EDT
***@wordsworth:~/debug# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.99.1 0.0.0.0 UG 1 0 0 br0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
***@wordsworth:~/debug# ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.99.3 netmask 255.255.255.0 broadcast 192.168.99.255
inet6 fe80::aaa1:59ff:fe26:9fbf prefixlen 64 scopeid 0x20<link>
ether a8:a1:59:26:9f:bf txqueuelen 1000 (Ethernet)
RX packets 115186 bytes 145420873 (138.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 74760 bytes 7391322 (7.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::aaa1:59ff:fe26:9fbf prefixlen 64 scopeid 0x20<link>
ether a8:a1:59:26:9f:bf txqueuelen 1000 (Ethernet)
RX packets 115132 bytes 147024486 (140.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 74911 bytes 7394972 (7.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 20 bytes 1258 (1.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20 bytes 1258 (1.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

sysdev: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::bc75:edff:fe8b:eac3 prefixlen 64 scopeid 0x20<link>
ether fe:39:85:3c:c0:12 txqueuelen 1000 (Ethernet)
RX packets 37 bytes 2228 (2.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10 bytes 1168 (1.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

veth6WN7NK: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc39:85ff:fe3c:c012 prefixlen 64 scopeid 0x20<link>
ether fe:39:85:3c:c0:12 txqueuelen 1000 (Ethernet)
RX packets 72 bytes 6440 (6.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 30 bytes 3245 (3.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vethI2G1MY: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc2d:13ff:fee1:ab85 prefixlen 64 scopeid 0x20<link>
ether fe:2d:13:e1:ab:85 txqueuelen 1000 (Ethernet)
RX packets 315 bytes 30748 (30.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 304 bytes 31539 (30.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vethPPSNFY: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fcc7:f9ff:fe9d:dea7 prefixlen 64 scopeid 0x20<link>
ether fe:c7:f9:9d:de:a7 txqueuelen 1000 (Ethernet)
RX packets 43 bytes 3795 (3.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 88 bytes 8308 (8.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vp0h: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc47:1aff:fec4:c96 prefixlen 64 scopeid 0x20<link>
ether fe:47:1a:c4:0c:96 txqueuelen 1000 (Ethernet)
RX packets 64 bytes 3210 (3.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 59 bytes 2888 (2.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

***@wordsworth:~/debug# cat /proc/sys/net/ipv4/ip_forward
1
***@wordsworth:~/debug# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.99.1 0.194 ms 0.138 ms 0.121 ms
2 my.isp.net.addr 13.030 ms 14.441 ms 15.925 ms
... elided for privacy ...
7 8.8.8.8 37.284 ms 26.002 ms 26.558 ms
***@wordsworth:~/debug# exit

Script done on Wed 01 May 2024 10:11:07 AM EDT

==== LXC configuration for SYSDEV1 / (192.168.99.32 / 192.168.55.1) ====
lxc.start.auto = 1
lxc.network.0.type = veth
lxc.network.0.flags = down
lxc.network.0.link = br0
lxc.network.1.type = veth
lxc.network.1.flags = down
lxc.network.1.name = vp0c
lxc.network.1.veth.pair = vp0h
lxc.network.2.type = veth
lxc.network.2.flags = down
lxc.network.2.name = eth1
lxc.network.2.link = sysdev
lxc.rootfs = /var/lib/lxc/sysdev1/rootfs
lxc.rootfs.backend = dir
lxc.utsname = sysdev1
lxc.mount = /var/lib/lxc/sysdev1/rootfs/etc/fstab
lxc.mount.entry = /dev/ppp dev/ppp none bind,create=file 0 0
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/sysdev1/rootfs
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm
lxc.cgroup.devices.allow = c 108:0 rwm
lxc.cap.drop = sys_module mknod mac_override mac_admin sys_time setfcap setpcap


==== LXC configuration for SYSDEV2 / (192.168.55.2) ====
lxc.network.0.type = veth
lxc.network.0.flags = down
lxc.network.0.name = eth0
lxc.network.0.link = sysdev
lxc.tty = 4
lxc.pts = 1024
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm
lxc.cap.drop = sys_module mknod mac_override mac_admin sys_time setfcap setpcap
lxc.rootfs = /var/lib/lxc/sysdev2/rootfs
lxc.rootfs.backend = dir
lxc.utsname = sysdev2
lxc.mount = /var/lib/lxc/sysdev2/fstab
--
Lew Pitcher
"In Skills We Trust"
Lew Pitcher
2024-05-01 17:12:52 UTC
Permalink
Post by Lew Pitcher
Hi, gals and guys
As I've mentioned before, I'm late to the container party
and starting to play around with LXC containers. This time,
my goal is to model a container setup for my internet-facing
server, to provide better security for its systems. More
specifically, I am trying to model a layout which puts
both my internet connection, and various internet-facing
services into containers such that the arrangement provides
a "bastion host" fronting a set of services running in a
"DMZ" network, sandwitched between firewalls.
Internet
|
=====(firewall)
Bastion Host
|
-----+----- ...
| |
=== === .....
me server
To this end, I've put together a couple of LXC containers,
one ("sysdev1") representing the "bastion host", and the
other ("sysdev2") representing a service within the "DMZ"
I should mention that, in this configuration, I /have not/
implemented any firewall rules. I want to get basic routing
working before I start complicating the data flow with a
firewall.

[snip]
--
Lew Pitcher
"In Skills We Trust"
Henrik Carlqvist
2024-05-02 05:15:19 UTC
Permalink
I should mention that, in this configuration, I /have not/ implemented
any firewall rules. I want to get basic routing working before I start
complicating the data flow with a firewall.
Maybe those firewall rules are exactly what you need. My guess is that
you are lacking NAT (IP masqueradning). When sending ping or any other
packet out from sysdev2 to internet those packages come from your private
IP address 192.168.55.2 but only the sysdev1 machine knows how to reach
back to that IP address. Your machine wordsworth with IP address
192.168.99.3 does not have a route back to 192.168.55.2. Either you will
need to implement NAT on sysdev1 or add a route on wordsworth (and
probably also implement a more cumbersome NAT in your real internet
facing router).

regards Henrik
Lew Pitcher
2024-05-02 13:21:26 UTC
Permalink
Hi, Henrik
Post by Henrik Carlqvist
I should mention that, in this configuration, I /have not/ implemented
any firewall rules. I want to get basic routing working before I start
complicating the data flow with a firewall.
Maybe those firewall rules are exactly what you need. My guess is that
you are lacking NAT (IP masqueradning). When sending ping or any other
packet out from sysdev2 to internet those packages come from your private
IP address 192.168.55.2 but only the sysdev1 machine knows how to reach
back to that IP address. Your machine wordsworth with IP address
192.168.99.3 does not have a route back to 192.168.55.2.
And, with that simple statement, you have solved my problem.
/Of course/, wordsworth needs a route back to sysdev2. And, in the
environment I'm trying to model, that route would be supplied by
a NAT rule on sysdev1.
Post by Henrik Carlqvist
Either you will
need to implement NAT on sysdev1 or add a route on wordsworth (and
probably also implement a more cumbersome NAT in your real internet
facing router).
Thanks, Henrik. You've hit the nail on the head.
And now, I'm off to make some firewall rules.
--
Lew Pitcher
"In Skills We Trust"
Lew Pitcher
2024-05-02 13:39:01 UTC
Permalink
Post by Lew Pitcher
Hi, Henrik
Post by Henrik Carlqvist
I should mention that, in this configuration, I /have not/ implemented
any firewall rules. I want to get basic routing working before I start
complicating the data flow with a firewall.
[snip]
Post by Lew Pitcher
Post by Henrik Carlqvist
Either you will
need to implement NAT on sysdev1 or add a route on wordsworth (and
probably also implement a more cumbersome NAT in your real internet
facing router).
Thanks, Henrik. You've hit the nail on the head.
And now, I'm off to make some firewall rules.
Yup. That fixed it.

Thanks again, Henrik
--
Lew Pitcher
"In Skills We Trust"
Loading...