Kubernetes’s IPTables Chains Are Not API
Author: Dan Winship (Red Hat) Some Kubernetes components (such as kubelet and kube-proxy) create iptables chains and rules as part of their operation. These chains were never intended to be part of any Kubernetes API/ABI guarantees, but some external components nonetheless make use of some of them (in particular, using KUBE-MARK-MASQ to mark packets as needing to be masqueraded).
Author: Dan Winship (Red Hat)
Some Kubernetes components (such as kubelet and kube-proxy) create
iptables chains and rules as part of their operation. These chains
were never intended to be part of any Kubernetes API/ABI guarantees,
but some external components nonetheless make use of some of them (in
KUBE-MARK-MASQ to mark packets as needing to be
As a part of the v1.25 release, SIG Network made this declaration explicit: that (with one exception), the iptables chains that Kubernetes creates are intended only for Kubernetes’s own internal use, and third-party components should not assume that Kubernetes will create any specific iptables chains, or that those chains will contain any specific rules if they do exist.
Then, in future releases, as part of KEP-3178, we will begin phasing
out certain chains that Kubernetes itself no longer needs. Components
outside of Kubernetes itself that make use of
KUBE-MARK-DROP, or other Kubernetes-generated iptables chains should
start migrating away from them now.
In addition to various service-specific iptables chains, kube-proxy
creates certain general-purpose iptables chains that it uses as part
of service proxying. In the past, kubelet also used iptables for a few
features (such as setting up
hostPort mapping for pods) and so it
also redundantly created some of the same chains.
However, with the removal of dockershim in Kubernetes in 1.24, kubelet now no longer ever uses any iptables rules for its own purposes; the things that it used to use iptables for are now always the responsibility of the container runtime or the network plugin, and there is no reason for kubelet to be creating any iptables rules.
iptables is still the default kube-proxy backend
on Linux, it is unlikely to remain the default forever, since the
associated command-line tools and kernel APIs are essentially
deprecated, and no longer receiving improvements. (RHEL 9
logs a warning if you use the iptables API, even via
Although as of Kubernetes 1.25 iptables kube-proxy remains popular, and kubelet continues to create the iptables rules that it historically created (despite no longer using them), third party software cannot assume that core Kubernetes components will keep creating these rules in the future.
Starting a few releases from now, kubelet will no longer create the
following iptables chains in the
KUBE-FIREWALL chain in the
filter table will no
longer have the functionality currently associated with
KUBE-MARK-DROP (and it may eventually go away entirely).
This change will be phased in via the
feature gate. That feature gate is available and can be manually
enabled for testing in Kubernetes 1.25. The current plan is that it
will become enabled-by-default in Kubernetes 1.27, though this may be
delayed to a later release. (It will not happen sooner than Kubernetes
What to do if you use Kubernetes’s iptables chains
(Although the discussion below focuses on short-term fixes that are still based on iptables, you should probably also start thinking about eventually migrating to nftables or another API).
If you use
If you are making use of the
KUBE-MARK-MASQ chain to cause packets
to be masqueraded, you have two options: (1) rewrite your rules to use
-j MASQUERADE directly, (2) create your own alternative “mark for
The reason kube-proxy uses
KUBE-MARK-MASQ is because there are lots
of cases where it needs to call both
-j DNAT and
-j MASQUERADE on
a packet, but it’s not possible to do both of those at the same time
DNAT must be called from the
chain (because it potentially changes where the packet will be routed
MASQUERADE must be called from
POSTROUTING (because the
masqueraded source IP that it picks depends on what the final routing
In theory, kube-proxy could have one set of rules to match packets in
OUTPUT and call
-j DNAT, and then have a second set
of rules to match the same packets in
POSTROUTING and call
-j MASQUERADE. But instead, for efficiency, it only matches them once,
OUTPUT, at which point it calls
-j DNAT and
-j KUBE-MARK-MASQ to set a bit on the kernel packet mark
as a reminder to itself. Then later, during
POSTROUTING, it has a
single rule that matches all previously-marked packets, and calls
-j MASQUERADE on them.
If you have a lot of rules where you need to apply both DNAT and
masquerading to the same packets like kube-proxy does, then you may
want a similar arrangement. But in many cases, components that use
KUBE-MARK-MASQ are only doing it because they copied kube-proxy’s
behavior without understanding why kube-proxy was doing it that way.
Many of these components could easily be rewritten to just use
separate DNAT and masquerade rules. (In cases where no DNAT is
occurring then there is even less point to using
just move your rules from
POSTROUTING and call
-j MASQUERADE directly.)
If you use
The rationale for
KUBE-MARK-DROP is similar to the rationale for
KUBE-MARK-MASQ: kube-proxy wanted to make packet-dropping decisions
alongside other decisions in the
KUBE-SERVICES chain, but you
can only call
-j DROP from the
filter table. So instead, it uses
KUBE-MARK-DROP to mark packets to be dropped later on.
In general, the approach for removing a dependency on
is the same as for removing a dependency on
kube-proxy’s case, it is actually quite easy to replace the usage of
KUBE-MARK-DROP in the
nat table with direct calls to
DROP in the
filter table, because there are no complicated interactions between
DNAT rules and drop rules, and so the drop rules can simply be moved
In more complicated cases, it might be necessary to “re-match” the
same packets in both
If you use Kubelet’s iptables rules to figure out
Components that manipulate host-network-namespace iptables rules from
inside a container need some way to figure out whether the host is
using the old
iptables-legacy binaries or the newer
binaries (which talk to a different kernel API underneath).
iptables-wrappers module provides a way for such components to
autodetect the system iptables mode, but in the past it did this by
assuming that Kubelet will have created “a bunch” of iptables rules
before any containers start, and so it can guess which mode the
iptables binaries in the host filesystem are using by seeing which
mode has more rules defined.
In future releases, Kubelet will no longer create many iptables rules, so heuristics based on counting the number of rules present may fail.
However, as of 1.24, Kubelet always creates a chain named
KUBE-IPTABLES-HINT in the
mangle table of whichever iptables
subsystem it is using. Components can now look for this specific chain
to know which iptables subsystem Kubelet (and thus, presumably, the
rest of the system) is using.
(Additionally, since Kubernetes 1.17, kubelet has created a chain
KUBE-KUBELET-CANARY in the
mangle table. While this chain
may go away in the future, it will of course still be there in older
releases, so in any recent version of Kubernetes, at least one of
KUBE-KUBELET-CANARY will be present.)
iptables-wrappers package has already been updated with this new
heuristic, so if you were previously using that, you can rebuild your
container images with an updated version of that.
The project to clean up iptables chain ownership and deprecate the old chains is tracked by KEP-3178.