Linux workstation and DNS servers

Dear Readers,

I am now blessed with having a full 100% Linux laptop at work. The laptop also travels between 2-3 clients with various network settings, all with their own DNS servers.

Usually, DHCP takes care of that and you haven’t got a care in the world.

Until now …. I now have a client with a few VPN’s that can turn on / off for various tasks and all of them push DNS settings to /etc/resolv.conf which makes other domains unreachable.

So the idea popped in my head: How to have different DNS servers for different domain-names on my laptop.

The answer was extremely stupid: install your own DNS server and use 127.0.0.1 as resolving!

See the following example:

[root@laptop:/root] # cat /etc/resolv.conf
domain first.domain.com
search second.domain.com third.domain.com
nameserver 127.0.0.1
nameserver 10.0.0.1

[root@laptop:/root] # tail -50 /etc/named.conf
zone "." IN {
        type forward;
        forwarders { 10.0.0.1; };  <<<<<<<<<<<<<<< This is the DNS server that DHCP gives you.
};

zone "second.domain.com" IN {
        type forward;
        forwarders { 192.168.1.1; }; <<<<<<<<<<<< Explicitly name a nameserver for second.domain.com

};

zone "third.domain.com" IN {
        type forward;
        forwarders { 192.168.2.4;  }; <<<<<<<<<<<< Explicitly name a nameserver for third.domain.com
};

Add more domains as you like. This solution will work for as many as 8 of them, because /etc/resolv.conf only supports 8 domain names in its search options.

Of course, you can add more than 8 but then you will have to always type the FQDN for domains 9 and higher.

Mark.

Openstack and Iscsi

Dear Readers,

My own datacenter had to go into ‘dark’ mode, because of imminent maintenance on the power-source.

This has led me to shutdown all VM’s on my network, then shut down the openstack compute nodes and finally openstack the physical machine. (in that order).

When the openstack nodes bootted again, one isolated test-compute/storage-node named yggdrasil wouldn’t boot a single VM… but to my horror, it ran the production IDM for all my hosts.

That meant none of my hosts were able to find eachother and slowly, all VM’s entered a rootshell because ISCSI couldn’t find the hard-drives…. woe me!

It turns out, i had run into 2 bugs at the same time.

Bug 1: The openstack cluster didn’t preserve its configuration ACL’s on the ISCSI device nodes.

Bug 2: My LVM devices were ‘discovered’ by the hardware node, so I couldn’t use targetcli to re-add them.

The path to discovering how this came to be.

First hint: I saw all the ACL’s were missing and therefore I got permission denied errors when trying to use iscsiadm to login to targets.

Since I had no comparison material, I had to find out the iscsi name first. Luckily it is saved in the file /etc/iscsi/initiatorname.iscsi.

So, I used the name and targetcli to re-add all the acl’s… now iscsiadm happily discovered a few nodes, but not all of them yet. Unfortunately, the discovered iscsi drives were not part of my IDM server so I had to continue debugging for now.

Then I noticed something horrible, one of my commands had failed because the logical volume (lvm) was “in-use” and therefore could not be added as a backstore from targetcli.

It cost me several days of googling until I finally hit the right answer.

lslbk …. YEP

The logical volume was supposed to be a harddisk, which contained another lvm configuration and stupidly enough, my hardware-node had activated the LVM configuration.

So I deactivated the lv with lvchange -a n /dev/system_vg/swap_lv.

I was lucky that my hardware node uses root_vg and not system_vg, or the mess might have been bigger!

And then I added a new filter content : ‘r|/dev/cinder-volume.*|’ to my lvm.conf to not have this happen again.

After this I could add the backing store, create the lun and acl. Add the thing to the portal and now iscsiadm would happily see all my luns. the others that were still missing had solved themselves automagically.

Booted my IDM and ‘lo and behold, all was coming up again and my world started spinning.

 

Mark.

 

What Satellite is still lacking in features that would make it even more excellent.

This post I am going to talk about what RHEL Satellite is still lacking and why that is a big deal. Implementing it would probably mean even heavier specs for the RHEL server thats running satellite, but I still think they would be valuable add-ons for Satellite.

  • Puppet Database (puppetdb).
  • Remote Execution (planned in 6.2)
  • Openscap maturity
  • Better Puppet modules management
  • Puppet updated to 4.0

Puppetdb

Using a puppet database, would mean that many things become available that are now a pain to implement in Puppet / Satellite.

I am talking about Puppet exported resources. If I want to use Satellite now to provision -say- a nagios monitoring host, I can’t make a puppet exported resources to allow for easy management of client hosts. I will have to either edit the nagios host manually (ouch!) or I will have to query the Foreman database for which hosts exist and what properties I need to assign them to complete my monitoring clients.

PuppetDB stores all facts and most recent report from its client-nodes. It also stores 14 days of history (a few config changes) which means that functionality can be chosen to be inside satellite, or inside PuppetDB.

Remote Execution

Remote execution needs little explaining, but after checking back with the Rhel Satellite 6.2 roadmap, I see Remote execution is already planned, so the point became moot :)

Openscap Maturity

What I mean by this point is, that the current Openscap implementation in Satellite is a bit sloppy. It doesn’t completely implement the Openscap editor tools, you need to know quite a lot about Openscap to write your own client policies and assign them to hosts. Of course you can download other people’s policies from the Internet and assign those, but then they aren’t auditted by your administration to make sure they fit company policy. Furthermore, Openscap currently crashes on me, leaving all reports on Satellite’s harddisk and not loaded in the GUI. Finding the faulty report is a pain and I have no idea why it is denied, as browsing it with a text-browser it seems fine.

Better puppet modules management.

Currently puppet modules are imported ‘as-is’ and assigned to content views and lifecycle management. However, despite the version-management of the content views, it is still unclear sometimes where puppet errors (duplicate class definitions , strange variables that appear and dissapear because of previous versions in other content views) come from. Whenever I encounter such an error, I try to push to the most recent puppet module verison on all my content views to be rid of old versions quickly.

Update to Puppet 4.0

Puppet 4.0 has many new features that are a boon to system administrators like myself.

  • Iteration over arrays / hashes
  • Bugs in facter solved (Primary ethernet interface anyone?)
  • Package collections. Read more about those on the Puppet BLOG
  • Better GIT integration
  • Robot 10000 (r10k)
  • HEREDOC support
  • Better TYPE support
  • Better performance on puppet server and clients (Faster catalog compile

 

Puppet & Satellite , the ENC explained.

Satellite has the capability to make facts available to puppet-clients through its ENC.

This process however is not documented well on Satellite’s , nor on Puppets documentation.

Take for example a class on the puppet-forge that requires some input for optional couplings:

maestrodev/archiva

This class has some optional parameters:

#
# $ldap = {
#   hostname      => 'ldap.example.com',
#   ssl           => true,
#   port          => '636',
#   dn            => 'o=example',
#   bind_dn       => "cn=admin,ou=system,o=example",
#   bind_password => "admin123",
#   admin_user    => "sysadmin",
#   guest_user    => "guest"
# }

Many blocks on the puppet forge have code like this, its a hash with values that should be enabled.

The puppet-initiates who have not yet fully grasped satellite, might implement it as following.

Create a new class, for instance:

  class oliekoets_archiva { 'archiva::ldap' :
    hostname => 'ldap.oliekoets.nl',
    ssl      => true,
    dn       => 'cn=users,cn=accounts,dc=oliekoets,dc=nl'
....

This however, is completely unnessecary in Satellite with the ENC.

Within satellite,  override the archiva::ldap variable and change its type from ‘string’ to ‘hash’

Now the hash-type accepts any valid json or yaml input, as long as it validates to a legal puppet object.

Here we go:

ldap: 
  hostname: 
    - ldap.oliekoets.nl
    - ldap2.oliekoets.nl
ssl: true
port: 636
dn: cn=users,cn=accounts,dc=oliekoets,dc=nl
....

Doublecheck by asking satellite to display its YAML enc:

---
classes:
  archiva:
    ldap:
      hostname:
        - ldap.oliekoets.nl
        - ldap2.oliekoets.nl
    ssl: true port: 636
    dn: cn=users,cn=accounts,dc=oliekoets,dc=nl
....

And thats all there is to it.

Mark.

 

TheForeman & Openstack

Quick note to boast about success!

Although my business isn’t all that big, I did find enough cash to finally start building up Oliekoets datacenter to a full-fledged private cloud.

Considering I have no wish yet to pay licensing fee’s of any kind, I Implemented it all with open-source freely available software.

CentOS , Foreman, Katello & Openstack.

Thanks all! you guys rock. Time to start building my virtual machines.

 

openstack-foreman

Puppet & Satellite part 2 , Server side facts!

Logo So today, I am talking about server-side facts.

These are facts a client can not and does not gather by itself. They are defined on the puppet server , which is usually the Satellite server running Foreman as well.

Unfortunately, Puppet within satellite does not allow the creation of server-side facts easily. You can set global & smart variables, which can become static and thus provide semi-facts, but nothing really sticks out as being convenient to use.

So far, I haven’t had the need to create server-side facts because my classes would reconfigure themselves easily through smart variables that can be overridden for specific servers, host groups or even globally.


 

Facts can be declared in a puppet class , but this would be a client-side fact.
Facts can also be declared as a global variable, this is a good example of a server-side fact

Server side facts can include for example: The hostname of your mail-relay, the hostnames of your IDM servers, the NTP/DNS servers, or any other fact really. Some facts are already pushed through by satellite itself: The life cycle environment of a host, the puppet environment, the organization, the host group of the host and a few others. These can be viewed by selecting a host and pushing the “Yaml” button. All custom facts you include yourself by means of global variables are also present in this Yaml display.

Lastly, facts can be declared from the puppet-master configuration directory. Since I never used them, I don’t know if Foreman displays them inside the web-interface or the hammer-interface of the host.

= Mark =

Puppet & Satellite part 1 , Facter!

LogoPuppet is a wonderful tool and as Foreman/RH Satellite now support puppet out of the box, it is time to explore. For the purpose of this blog post, I shall assume
RH Satellite == Foreman!

The first thing to be noted, is that puppet is severely under-documented in the redhat and foreman documentation. Little is provided and even those little things are creating more questions than answering them. I will write a few blog posts on this very subject, of which this is post 1.

First topic in this series: Facts!.

The Foreman project is adding facts to the satellite server through facter , which is part of puppet. But what if these facts aren’t complete. Its all nice and dandy to know standard stuff like a hosts ip-address and hostname and cpu and stuff, but what if we need to know for instance , the names of the IDM servers for the NTP configuration which has been integrated in Satellite. You won’t find those anywhere.


First lets apply a few logic rules to determine whether we need this new fact, or would rather use it as a variable to be declared in some class::param concept.

When to use a fact in my opinion

– It never changes! This seems obvious, but I have seen administrators getting this wrong, I have seen guys write ruby code to change a fact almost every run! Don’t do this people, if something isn’t solid, use a class:param and not a fact! (for instance, the system time :) this is not a fact guys, its completely variable!, it changes as fast as the clock resolution of your computer!)

The fact will be either master-side , or client-side compiled. Facts created on the puppet-master for instance might include the ntp-servers to use. Client-side facts might be dependant of the environment , like the nfs-homeserver of your cloud environment. Production might have a different one than Development but other than that it is static, it can be put into fact.

Next post: Server-side facts.

RHEL Satellite access

Today I am going to talk about (remote) access with satellite.

Apparently, there are a few things that you must know in order to get stuff working correctly.

First of all: Red Hat Identity Manager <-> Satellite coupling for user accounts.
When you create the coupling as an external LDAP source in satellite, by default users get put in the anonymous group with very little rights within satellite. Luckily you can also provide a “group” DN for Identity servers which can then be used to assign groups in satellite.


 

So create a user group (for instance : Admins) in the satellite user interface. Then in the third (external group) tab , assign a coupling between a redhat identity manager (IDM) group and the local admin group. The source however, must be set to “External” instead of your identity server, I am not sure if this is a bug or works as designed. Now, when users login who have the correct LDAP group, will automaticly be added to the new Admins group on satellite. Now you can assign rights (or even check the full admin checkbox) to the Admin usergroup and the remote access is done.


Now a short paragraph about local webinterface access as the default admin account:

When satellite needs reconfiguring, or reinstalling, Red Hat notes that the admin password gets reset to a default password and you will simply have to change it again. This is not entirely true. You can put the password of the default admin user in /etc/katello-installer/answers.katello-installer.yaml, but doing so is a security risk according to some people. I am noting that if you have root-access , the security risk of this file is negligant, because you can simply run katello-installer without any arguments and it will printout the admin password on the console after a succesful completion.

– Mark.

 

Firewall rules incomplete when using Autodiscovery with RHEL Satellite 6.1

There are some things not yet in the RedHat manual concerning implementing discovered hosts provisioning through use of the discovery image in Red Hat Satellite Server.

First of all, the FDI is also running a very limited foreman-proxy server, so to get full functionality you also need to open port 8443 on the subnet where clients are provisioned in the firewalls. Failure to do so will prevent a discovered client to reboot as instructed by satellite and you will have to push the button by hand.

So if your server runs on 10.0.0.1 and is provisioning for clients on subnet 10.2.0.0/24 with a firewall in between, you need to “allow” traffic from 10.0.0.1 any port , to 10.2.0.0/24 destination port 8443/TCP , as well.

Also not widely documented yet, you can enable SSH login on the discovery image with 2 kernel-boot options , add them as usual in the satellite global PXEboot file.

fdi.ssh=1 and fdi.rootpw=welcome  , which enables ssh and sets the root password to welcome.

– Mark.

Fonts, Fonts, Fonts

During my work, I spend a lot of the time reading a screen and interpreting the stuff that is on it. A good solid font is definitely a must-have for me.

I used to be content with a font called “Courier New 10pt”. However , I came across this little gem :

http://www.donationcoder.com/Software/Jibz/Dina/index.html

This font “Dina” is freeware, meaning anyone can use it.

The author used the following guidelines designing this font:

  • Mono spaced
  • Should be easy to distinguish between `j i l 1 I`
  • Should be easy to distinguish between `o O 0`
  • Operators should line up horizontally `- + * =`
  • Brackets should line up horizontally and vertically `< ( { [ ] } ) >`
  • Punctuation should be clear `., :; ‘ “`
  • Symbols used in programming languages should look right `& @ % $ #`
  • No other characters that look too similar `gqy z2Z s5S 8B CG6 DO uv`
  • Still has to be pleasant to read passages of text

If you use it and like it, please donate to the author who spent time to create this gem.