Cannot get working with custom JSON objects (Documentation confusion)

[jpSimkins]

Cookbook version

2.7.0

Chef-client version

14.9.13

Platform Details

CentOS7

Scenario:

Not sure if it's just me but the instructions are not very clear but I gave it the old college try...

What I am trying to do is build a recipe that allows me to use data_bags and environments to specify rules. Most of the nodes are going to be CentOS 7 but I have a few Ubuntu 16 too (maybe different versions pending third-party software). Most nodes will have 2 NICs, 1 with zone public and another with trusted.

During development, I used rspec and everything worked as expected. When I went to Test Kitchen, well, that's when I found out nothing was working. So I shelled into the node and realized no rules were being applied.

Ideally, I first want the default zone to be :public.
Then I wanted to apply rules to each zone from the json objects. For this ticket, I only have public defined but this is still problematic.
When testing with rspec, it all looked good.
When I used Test Kitchen, I can sometimes get the zone to be public. I can also see the rules in iptables but they don't seem to be zone specific?

Hopefully I am just doing something stupid or am not clear on this part. I am hoping for a push in the right direction here... Or at least clarify the documentation a bit because it seems incomplete and no working examples.

Thanks for your time.

An example of an environment (dev):

{
  "name": "dev",
  "description": "DEV Environment for Nodes",
  "chef_type": "environment",
  "json_class": "Chef::Environment",
  "default_attributes": {
    "oly": {
      "environment": "dev",
      "type" : "node",
      "firewall": {
        "status": "enabled",
        "zones": {
          "public": {
            "22": {
              "private_ip_1": "10.0.0.0/8",
              "private_ip_2": "172.16.0.0/12",
              "private_ip_3": "192.168.0.0/16",
              "private_ip_4": "169.254.0.0/16",
              "private_ip_5": "100.64.0.0/10"
            }
          }
        }
      }
    }
  },
  "cookbook_versions": {
    "oly-client": "= 4.0.0"
  }
}

The above environment has a firewall zone config that opens up port 22 for all private IP addresses.

An example of a data_bag (firewall:global) is:

{
  "id": "global",
  "zones": {
    "public": {
      "22": {
          "office_1": "1.1.1.1/32",
          "office_2": "2.2.2.2/32",
          "office_3": "3.3.3.3/32",
          "office_4": "4.4.4.4/32",
          "office_5": "5.5.5.5/32"
      }
    }
  }
}

Ideally, this allows global rules to be applied to the recipe.

The cookbook I am working on (It's been modified a bit from different tickets and google results, this is my current version):

#
# Cookbook:: oly-client
# Recipe:: firewall
# 
# TODO: Create a method to optimize code (code repetition is real here)

# Fetch firewall settings
_firewallSettings = node['oly']['firewall']

# Make sure we have firewall settings and that they are enabled
if (!_firewallSettings.to_a.empty? && _firewallSettings.key?("status") && 'enabled' == _firewallSettings['status'].downcase)
  # include the base firewall recipe
  include_recipe "firewall::default"

  # Enable platform default firewall and set default zone
  firewall 'default' do
    enabled_zone :public
    action :install
  end

  # START global firewall rules
  _globalFirewallRules = data_bag_item('firewall', 'global')
  if (_globalFirewallRules && _globalFirewallRules.key?("zones"))

    # Loop over each firewall zone and build rules from data
    _globalFirewallRules['zones'].each do |_zone, _zoneData|

      # Ensure we have zone data
      if (_zoneData)

        # Ensure the firewall is installed for the zone
        firewall "#{_zone}" do
          enabled_zone "#{_zone}".to_sym
          action :install
        end

        # Process rules for firewall
        _zoneData.each do |_port, _portRules|
          # Verify rules exist
          if (_portRules)
            # Build rules
            _portRules.each do |_ipComment, _ipAddress|

              # Define rule
              firewall_rule "#{_zone} - #{_port}: #{_ipComment} - #{_ipAddress}" do
                firewall_name "#{_zone}"
                protocol :tcp
                port _port.to_i
                source "#{_ipAddress}"
                direction :in
                command :allow
                action :create
              end

            end

          end

        end

      end

    end

  end
  # END global firewall rules

  # Check if environment has any zones configured
  if (_firewallSettings.key?("zones"))

    # Loop over each firewall zone and build rules from data
    _firewallSettings['zones'].each do |_zone, _zoneData|

      # Ensure we have zone data
      if (_zoneData)

        # Ensure the firewall is installed for the zone (in case global zones does not include)
        firewall "#{_zone}" do
          enabled_zone "#{_zone}".to_sym
          action :install
        end

        # Process rules for firewall
        _zoneData.each do |_port, _portRules|
          # Verify rules exist
          if (_portRules)
            # Build rules
            _portRules.each do |_ipComment, _ipAddress|

              # Define rule
              firewall_rule "#{_zone} - #{_port}: #{_ipComment} - #{_ipAddress}" do
                firewall_name "#{_zone}"
                protocol :tcp
                port _port.to_i
                source "#{_ipAddress}"
                direction :in
                command :allow
                action :create
              end

            end

          end

        end

      end

    end

  end
  # END environment firewall rules

  # TODO Add logic for custom rules (with search capabilites, like users - Did not do yet as this is edge case if needed at all)

else
  # Firewall is disabled unless explicitly enabled
  include_recipe 'firewall::disable_firewall'
end

My rspec test (replaced IPs but should work the same):

#
# Cookbook:: oly-client
# Spec:: default
#
# Copyright:: 2017, The Authors, All Rights Reserved.

require 'spec_helper'

describe 'oly-client::firewall' do

  context 'on CentOS 7 Latest' do

    let(:chef_run) do
      ChefSpec::SoloRunner.new(platform: 'centos', version: '7') do |node|
        
        # Build node attributes for tests
        node.normal['oly']['firewall']['status'] = "enabled"
        node.normal['oly']['firewall']['zones'] = {
          "public": {
            "22": {
              "private_ip_1": "10.0.0.0/8",
              "private_ip_2": "172.16.0.0/12",
              "private_ip_3": "192.168.0.0/16",
              "private_ip_4": "169.254.0.0/16"
            }
          },
          "trusted": {
            "22": {
              "private_ip_5": "100.64.0.0/10"
            }
          }
        }

        # Firewall rules
        node.normal['firewall']['allow_icmp'] = true
        node.normal['firewall']['allow_ssh'] = true
        node.normal['firewall']['allow_winrm'] = false
        node.normal['firewall']['allow_mosh'] = false

      end.converge(described_recipe)
    end

    # Stub databags
    before do
      stub_data_bag('firewall').and_return(['global'])
      stub_data_bag_item('firewall', 'global').and_return({
        "id": "global",
        "zones": {
          "public": {
            "22": {
                  "office_1": "1.1.1.1/32",
                  "office_2": "2.2.2.2/32",
                  "office_3": "3.3.3.3/32"
            }
          },
          "trusted": {
            "22": {
              "office_1": "1.1.1.1/32",
              "office_3": "3.3.3.3/32",
              "office_4": "4.4.4.4/32",
              "office_5": "5.5.5.5/32"
            }
          }
        }
      })
    end

    it 'include the recipe to enable firewall' do
      expect(chef_run).to include_recipe('firewall::default')
    end

    it 'enables the firewall' do
      expect(chef_run).to install_firewall('public')
      expect(chef_run).to install_firewall('trusted')
    end
    
    it 'creates some rules' do
      _rules = [
        "allow loopback", 
        "allow icmp", 
        "allow world to ssh", 
        "established",
        "ipv6_icmp",
        "public - 22: private_ip_1 - 10.0.0.0/8",
        "public - 22: private_ip_2 - 172.16.0.0/12",
        "public - 22: private_ip_3 - 192.168.0.0/16",
        "public - 22: private_ip_4 - 169.254.0.0/16",
        "trusted - 22: private_ip_5 - 100.64.0.0/10",
        "public - 22: office_1 - 1.1.1.1/32",
        "public - 22: office_2 - 2.2.2.2/32",
        "public - 22: office_3 - 3.3.3.3/32",
        "trusted - 22: office_1 - 1.1.1.1/32",
        "trusted - 22: office_3 - 3.3.3.3/32",
        "trusted - 22: office_4 - 4.4.4.4/32",
        "trusted - 22: office_5 - 5.5.5.5/32"
      ]

      _rules.each do |r|
        expect(chef_run).to create_firewall_rule(r)
      end
    end
    
    
    it 'not to creates some rules' do
      _rules = [
        "allow world to winrm", 
        "allow world to mosh",
        "public - 22: office_4 - 4.4.4.4/32",
        "public - 22: office_5 - 5.5.5.5/32",
        "trusted - 22: office_2 - 2.2.2.2/32"
      ]

      _rules.each do |r|
        expect(chef_run).to_not create_firewall_rule(r)
      end
    end

  end

end

I do see output like it is being applied when running Test Kitchen:
(removed some personal IPs from the output)

   Recipe: oly-client::firewall
         * firewall[public] action restart
           * file[/etc/sysconfig/firewalld-chef.rules] action create
             - update content in file /etc/sysconfig/firewalld-chef.rules from fa85ee to 2ad8b9
             --- /etc/sysconfig/firewalld-chef.rules    2019-01-22 21:15:19.759654943 +0000
             +++ /etc/sysconfig/.chef-firewalld-chef20190122-16449-1dmtitl.rules        2019-01-22 21:16:26.545532813 +0000
             @@ -1,2 +1,22 @@
             -# created by chef to allow service to start
             +# position 50
             +firewall-cmd --direct --add-rule ipv4 filter INPUT 50 -s 10.0.0.0/8 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_1 - 10.0.0.0/8' -j ACCEPT
             +firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 50 -s 10.0.0.0/8 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_1 - 10.0.0.0/8' -j ACCEPT
             +firewall-cmd --direct --add-rule ipv4 filter INPUT 50 -s 172.16.0.0/12 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_2 - 172.16.0.0/12' -j ACCEPT
             +firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 50 -s 172.16.0.0/12 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_2 - 172.16.0.0/12' -j ACCEPT
             +firewall-cmd --direct --add-rule ipv4 filter INPUT 50 -s 192.168.0.0/16 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_3 - 192.168.0.0/16' -j ACCEPT
             +firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 50 -s 192.168.0.0/16 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_3 - 192.168.0.0/16' -j ACCEPT
             +firewall-cmd --direct --add-rule ipv4 filter INPUT 50 -s 169.254.0.0/16 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_4 - 169.254.0.0/16' -j ACCEPT
             +firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 50 -s 169.254.0.0/16 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_4 - 169.254.0.0/16' -j ACCEPT
             +firewall-cmd --direct --add-rule ipv4 filter INPUT 50 -s 100.64.0.0/10 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_5 - 100.64.0.0/10' -j ACCEPT
             +firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 50 -s 100.64.0.0/10 -p tcp -m tcp -m multiport --dports 22 -m comment --comment 'public - 22: private_ip_5 - 100.64.0.0/10' -j ACCEPT
             - restore selinux security context
           * service[firewalld] action enable (up to date)
           * service[firewalld] action start (up to date)

It seems the rules are in iptables (didn't realize firewalld was a wrapper for iptables?) but the rules don't appear to be locked to a zone. Meaning, when I use on a node with 2 NICs, the rules are applied to both NICs (they are in 2 different zones)

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_drop  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_drop  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_drop  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_drop  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_direct (1 references)
target     prot opt source               destination

Chain FWDI_drop (2 references)
target     prot opt source               destination
FWDI_drop_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_drop_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_drop_allow  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain FWDI_drop_allow (1 references)
target     prot opt source               destination

Chain FWDI_drop_deny (1 references)
target     prot opt source               destination

Chain FWDI_drop_log (1 references)
target     prot opt source               destination

Chain FWDI_public (0 references)
target     prot opt source               destination
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain FWDI_public_allow (1 references)
target     prot opt source               destination

Chain FWDI_public_deny (1 references)
target     prot opt source               destination

Chain FWDI_public_log (1 references)
target     prot opt source               destination

Chain FWDO_drop (2 references)
target     prot opt source               destination
FWDO_drop_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_drop_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_drop_allow  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain FWDO_drop_allow (1 references)
target     prot opt source               destination

Chain FWDO_drop_deny (1 references)
target     prot opt source               destination

Chain FWDO_drop_log (1 references)
target     prot opt source               destination

Chain FWDO_public (0 references)
target     prot opt source               destination
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0

Chain FWDO_public_allow (1 references)
target     prot opt source               destination

Chain FWDO_public_deny (1 references)
target     prot opt source               destination

Chain FWDO_public_log (1 references)
target     prot opt source               destination

Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_drop    all  --  0.0.0.0/0            0.0.0.0/0
IN_drop    all  --  0.0.0.0/0            0.0.0.0/0

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain INPUT_direct (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  10.0.0.0/8           0.0.0.0/0            tcp multiport dports 22 /* public - 22: private_ip_1 - 10.0.0.0/8 */
ACCEPT     tcp  --  172.16.0.0/12        0.0.0.0/0            tcp multiport dports 22 /* public - 22: private_ip_2 - 172.16.0.0/12 */
ACCEPT     tcp  --  192.168.0.0/16       0.0.0.0/0            tcp multiport dports 22 /* public - 22: private_ip_3 - 192.168.0.0/16 */
ACCEPT     tcp  --  169.254.0.0/16       0.0.0.0/0            tcp multiport dports 22 /* public - 22: private_ip_4 - 169.254.0.0/16 */
ACCEPT     tcp  --  100.64.0.0/10        0.0.0.0/0            tcp multiport dports 22 /* public - 22: private_ip_5 - 100.64.0.0/10 */

Chain IN_drop (2 references)
target     prot opt source               destination
IN_drop_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_drop_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_drop_allow  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0

Chain IN_drop_allow (1 references)
target     prot opt source               destination

Chain IN_drop_deny (1 references)
target     prot opt source               destination

Chain IN_drop_log (1 references)
target     prot opt source               destination

Chain IN_public (0 references)
target     prot opt source               destination
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0

Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination

Chain IN_public_log (1 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

Expected Result:

I expect the default zone to be public and my ips to be whitelisted. I expected my rules to be locked to the specified zone and be permanent.

Actual Result:

Default zone stays drop and no rules are applied. Sometimes, through many edits, I can get the zone to be public but no rules are applied. Just seems some clarification or better examples in the documentation would help a lot.

[martinb3]

Hi there -- it's true, firewalld is a wrapper on the same kernel structures as iptables ;)

While there is a zone attribute on the firewall resource, I haven't tested this cookbook with multiple NICs or multiple/custom zones. The operative code for writing rules, that we'd need to make zone-aware, is at https://github.com/chef-cookbooks/firewall/blob/master/libraries/helpers_firewalld.rb#L77.

It looks like we'd need to add a zone parameter to the firewall_rule resource, and then in that code I linked, add support for --zone=public as part of the command.

This cookbook was meant to be a first approximation to the general idea of a firewall across a bunch of different OSes and firewall implementations, but using multiple zones is stretching it. If it saves you some pain, there may be a firewalld-only cookbook that has deeper support for firewalld-specific concepts.

I'm happy to review & test a PR if you're interested in taking a shot at support for this in the firewall cookbook 👍

[jpSimkins]

Thanks for your feedback. I completely understand. I was trying to use this cookbook due to being more abstract but I can see how my needs are a bit tightly coupled to firewalld. If it's just modifying that method to add in the --zone attribute and extending the firewall_rule resource, I can take a stab at that.

My last part for this is some clarification then. I assume, that I only need 1:

  firewall "default" do
    action :install
  end

In my loop I am doing:

        firewall "#{_zone}" do
          enabled_zone "#{_zone}".to_sym
          action :install
        end

but that seems not necessary at all. I just need to loop over the rules and apply them Adding the zone would give me the result I want.

By using enabled_zone I can force the default zone to be public so there is no need to build a new firewall for each zone.

Just trying to clear up some confusion. To me, it seems I only need 1 declaration of firewall.

Thanks again