User Tools

Site Tools


soft:node-software

====== NODES (SOFTWARE) ====== ====== Introduction ======

This document describes the CLOMMUNITY 1) software used. The CLOMMUNITY nodes use CONFINE NODE SOFTWARE (CNS) 2) plus the support to be managed by Open Nebula 3) with the libvirt library 4). To provide Services for the project, the system uses the Guifi-Community-Distro.

Libvirt in OpenWRT

Introduction

We report on the steps we took in order to integrate Libvirt in OpenWRT.

(Note: This line of work did not find further application later in the project when the Cloudy distribution started to focus on mini-PCs. The results of this work, however, should remain visible, since it may open interesting opportunities in the future)

Libvirt is collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management. These software pieces include an API library, a daemon (libvirtd), and a command line utility (virsh).

Open Nebula uses the libvirt API to manage multiple different virtualization providers/hypervisors like KVM, Xen, VMWare ESX, etc., so we have to port the library to an OpenWRT package 5).

Preparing the package

In order to create a package in OpenWRT firstly we have to prepare the directory and the files to make it.

  • Make a directory inside the confine packages':
$ cd ~/confine/packages/confine/
$ mkdir libvirt

Then we have to add two files inside the package directory:

Files

Makefile

The Makefile contains the sources to download the code, the dependences and the instructions to compile and create the package.

(You can find updated files at confine repository 6). Test using v0.9.11)

include $(TOPDIR)/rules.mk
 
PKG_NAME:=libvirt
#PKG_VERSION:=1.0.4
PKG_VERSION:=0.10.2
#PKG_VERSION:=0.9.11
PKG_RELEASE:=3
 
PKG_BUILD_DIR:=$(BUILD_DIR)/$(PKG_NAME)-$(PKG_VERSION)
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=http://libvirt.org/sources/
#v1.0.4
#PKG_MD5SUM:=97166bc42d7cacb037923907abe656ab
#v0.10.2
PKG_MD5SUM:=a5e50860d9da238ba270b528411c4a0d
#v0.9.11
#PKG_MD5SUM:=6e9fab115075a8fd21d9bd0d7e558a52
 
PKG_INSTALL:=1
 
 
include $(INCLUDE_DIR)/package.mk
 
define Package/libvirt
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=Libvirt Virtualization API
  URL:=http://libvirt.org
  DEPENDS:= +gnutls-utils +libgnutls +libgnutls-extra +libgnutls-openssl +certtool \
            +libdevmapper +libnl +libgcrypt +librpc +libxml2 +dmidecode \
            +gzip +tar
 
#TO DO: Check and remove unneeded dependencies
 
endef
 
TARGET_CFLAGS += -fno-stack-protector
 
CONFIGURE_VARS += BLKID_LIBS="-lblkid -luuid -lm" \
                  gl_cv_warn__fstack_protector_all=no \
                  gl_cv_warn_c__fstack_protector_all=no \
 
 
define Package/libvirt/description
	Libvirt is an open source API, daemon and management tool for managing platform virtualization.
endef
 
define Build/Configure
	$(call Build/Configure/Default,--with-yajl=no --with-macvtap=no --with-lxc=yes)
endef
 
define Package/libvirt/install
	$(CP) $(PKG_INSTALL_DIR)/etc $(1)/
	$(CP) $(PKG_INSTALL_DIR)/usr $(1)/
 
#	$(CP) $(PKG_INSTALL_DIR)/var $(1)/ # Cannot do so use an init script
	$(INSTALL_DIR) $(1)/etc
	$(INSTALL_DIR) $(1)/etc/init.d
	$(INSTALL_BIN) ./files/etc/init.d/libvirt-init $(1)/etc/init.d/
endef
 
$(eval $(call BuildPackage,libvirt))

Init script

Because of /var directory in OpenWRT is placed in a temporary filesystem we have to create some directories of the library at each system startup with an init script.

Enter package directory:

$ cd ~/confine/packages/confine/libvirt/
$ mkdir -p files/etc/init.d/

Create a file inside the directory with the following code, and make it executable:

$ chmod +x libvirt-init

libvirt-init:

#!/bin/sh /etc/rc.common
 
START=80
 
boot() {
	start
}
 
start() {
	# Create /var/cache/libvirt if not exists
	if [ ! -d  /var/cache/libvirt ]; then
		mkdir -p /var/cache/libvirt/qemu
	fi
 
	# Create /var/lib/libvirt if not exists
	if [ ! -d  /var/lib/libvirt ]; then
		mkdir -p /var/lib/libvirt/boot
		mkdir -p /var/lib/libvirt/dnsmasq
		mkdir -p /var/lib/libvirt/filesystems
		mkdir -p /var/lib/libvirt/images
		mkdir -p /var/lib/libvirt/lockd/files
		mkdir -p /var/lib/libvirt/lxc
		mkdir -p /var/lib/libvirt/network
		mkdir -p /var/lib/libvirt/qemu
		mkdir -p /var/lib/libvirt/uml
	fi
 
	# Create /var/log/libvirt if not exists
	if [ ! -d  /var/log/libvirt ]; then
	        mkdir -p /var/log/libvirt/lxc
        	mkdir -p /var/log/libvirt/qemu
        	mkdir -p /var/log/libvirt/uml
	fi
 
	# Create /var/run/libvirt if not exists
	if [ ! -d  /var/run/libvirt ]; then
	        mkdir -p /var/run/libvirt/lockd
        	mkdir -p /var/run/libvirt/lxc
        	mkdir -p /var/run/libvirt/network
        	mkdir -p /var/run/libvirt/qemu
        	mkdir -p /var/run/libvirt/uml
	fi
 
	# Starting libvirtd daemon
	echo "Starting libvirt daemon..."
	exec /usr/sbin/libvirtd -d
}
 
stop() {
	echo "Stoping libvirt daemon..."
	start-stop-daemon -K -p /var/run/libvirtd.pid
}

Compiling the package

Before compiling

In order to compile the libvirt package we need to add some modifications to the system to be able tu run properly the library.

Libvirt needs the support of the CHECKSUM iptable's option.

Iptables recent version with CHECKSUM option

Firstly we have to compile the newer version of iptables that has this feature

  • Enter into the iptables package directory:
$ cd openwrt/package/iptables/
  • Remove the 'patches' directory:
$ rm -rf patches
  • Add/rewrite the Makefile with the following:
#
# Copyright (C) 2006-2012 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#
 
include $(TOPDIR)/rules.mk
include $(INCLUDE_DIR)/kernel.mk
 
PKG_NAME:=iptables
PKG_VERSION:=1.4.18
PKG_RELEASE:=4
 
PKG_MD5SUM:=a819199d5ec013b82da13a8ffbba857e
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.bz2
PKG_SOURCE_URL:=http://www.netfilter.org/projects/iptables/files \
	ftp://ftp.be.netfilter.org/pub/netfilter/iptables/ \
	ftp://ftp.de.netfilter.org/pub/netfilter/iptables/ \
	ftp://ftp.no.netfilter.org/pub/netfilter/iptables/
 
PKG_FIXUP:=autoreconf
PKG_INSTALL:=1
PKG_BUILD_PARALLEL:=1
 
ifneq ($(CONFIG_EXTERNAL_KERNEL_TREE),"")
PATCH_DIR:=
endif
 
include $(INCLUDE_DIR)/package.mk
ifeq ($(DUMP),)
  -include $(LINUX_DIR)/.config
  include $(INCLUDE_DIR)/netfilter.mk
  STAMP_CONFIGURED:=$(strip $(STAMP_CONFIGURED))_$(shell $(SH_FUNC) grep 'NETFILTER' $(LINUX_DIR)/.config | md5s)
endif
 
 
define Package/iptables/Default
  SECTION:=net
  CATEGORY:=Network
  SUBMENU:=Firewall
  URL:=http://netfilter.org/
endef
 
define Package/iptables/Module
$(call Package/iptables/Default)
  DEPENDS:=iptables $(1)
endef
 
define Package/iptables
$(call Package/iptables/Default)
  TITLE:=IPv4 firewall administration tool
  MENU:=1
  DEPENDS+= +kmod-ipt-core +libip4tc +libxtables +libiptc +libip6tc
endef
 
define Package/iptables/description
IPv4 firewall administration tool.
 
 Matches:
  - icmp
  - tcp
  - udp
  - comment
  - limit
  - mac
  - multiport
 
 Targets:
  - ACCEPT
  - DROP
  - REJECT
  - LOG
  - TCPMSS
 
 Tables:
  - filter
  - mangle
 
endef
 
define Package/iptables-mod-conntrack-extra
$(call Package/iptables/Module, +kmod-ipt-conntrack-extra)
  TITLE:=Extra connection tracking extensions
endef
 
define Package/iptables-mod-conntrack-extra/description
Extra iptables extensions for connection tracking.
 
 Matches:
  - connbytes
  - connmark
  - recent
  - helper
 
 Targets:
  - CONNMARK
 
endef
 
define Package/iptables-mod-filter
$(call Package/iptables/Module, +kmod-ipt-filter)
  TITLE:=Content inspection extensions
endef
 
define Package/iptables-mod-filter/description
iptables extensions for packet content inspection.
Includes support for:
 
 Matches:
  - layer7
  - string
 
endef
 
define Package/iptables-mod-ipopt
$(call Package/iptables/Module, +kmod-ipt-ipopt)
  TITLE:=IP/Packet option extensions
endef
 
define Package/iptables-mod-ipopt/description
iptables extensions for matching/changing IP packet options.
 
 Matches:
  - dscp
  - ecn
  - length
  - mark
  - statistic
  - tcpmss
  - time
  - unclean
  - hl
 
 Targets:
  - DSCP
  - CLASSIFY
  - ECN
  - MARK
  - HL
 
endef
 
define Package/iptables-mod-ipsec
$(call Package/iptables/Module, +kmod-ipt-ipsec)
  TITLE:=IPsec extensions
endef
 
define Package/iptables-mod-ipsec/description
iptables extensions for matching ipsec traffic.
 
 Matches:
  - ah
  - esp
  - policy
 
endef
 
define Package/iptables-mod-ipset
$(call Package/iptables/Module,)
  TITLE:=IPset iptables extensions
endef
 
define Package/iptables-mod-ipset/description
IPset iptables extensions.
 
 Matches:
  - set
 
 Targets:
  - SET
 
endef
 
define Package/iptables-mod-nat-extra
$(call Package/iptables/Module, +kmod-ipt-nat-extra)
  TITLE:=Extra NAT extensions
endef
 
define Package/iptables-mod-nat-extra/description
iptables extensions for extra NAT targets.
 
 Targets:
  - MIRROR
  - NETMAP
  - REDIRECT
endef
 
define Package/iptables-mod-ulog
$(call Package/iptables/Module, +kmod-ipt-ulog)
  TITLE:=user-space packet logging
endef
 
define Package/iptables-mod-ulog/description
iptables extensions for user-space packet logging.
 
 Targets:
  - ULOG
 
endef
 
define Package/iptables-mod-hashlimit
$(call Package/iptables/Module, +kmod-ipt-hashlimit)
  TITLE:=hashlimit matching
endef
 
define Package/iptables-mod-hashlimit/description
iptables extensions for hashlimit matching
 
 Matches:
  - hashlimit
 
endef
 
define Package/iptables-mod-iprange
$(call Package/iptables/Module, +kmod-ipt-iprange)
  TITLE:=IP range extension
endef
 
define Package/iptables-mod-iprange/description
iptables extensions for matching ip ranges.
 
 Matches:
  - iprange
 
endef
 
define Package/iptables-mod-extra
$(call Package/iptables/Module, +kmod-ipt-extra)
  TITLE:=Other extra iptables extensions
endef
 
define Package/iptables-mod-extra/description
Other extra iptables extensions.
 
 Matches:
  - condition
  - owner
  - physdev (if ebtables is enabled)
  - pkttype
  - quota
 
endef
 
define Package/iptables-mod-led
$(call Package/iptables/Module, +kmod-ipt-led)
  TITLE:=LED trigger iptables extension
endef
 
define Package/iptables-mod-led/description
iptables extension for triggering a LED.
 
 Targets:
  - LED
 
endef
 
define Package/iptables-mod-tproxy
$(call Package/iptables/Module, +kmod-ipt-tproxy)
  TITLE:=Transparent proxy iptables extensions
endef
 
define Package/iptables-mod-tproxy/description
Transparent proxy iptables extensions.
 
 Matches:
  - socket
 
 Targets:
  - TPROXY
 
endef
 
define Package/iptables-mod-tee
$(call Package/iptables/Module, +kmod-ipt-tee)
  TITLE:=TEE iptables extensions
endef
 
define Package/iptables-mod-tee/description
TEE iptables extensions.
 
 Targets:
  - TEE
 
endef
 
define Package/iptables-mod-u32
$(call Package/iptables/Module, +kmod-ipt-u32)
  TITLE:=U32 iptables extensions
endef
 
define Package/iptables-mod-u32/description
U32 iptables extensions.
 
 Matches:
  - u32
 
endef
 
define Package/ip6tables
$(call Package/iptables/Default)
  DEPENDS:=+kmod-ip6tables +libip6tc +libxtables +libip4tc
  CATEGORY:=IPv6
  TITLE:=IPv6 firewall administration tool
  MENU:=1
endef
 
define Package/libiptc
$(call Package/iptables/Default)
  SECTION:=libs
  CATEGORY:=Libraries
  DEPENDS:=+libip4tc +libip6tc
  TITLE:=IPv4/IPv6 firewall - shared libiptc library (compatibility stub)
endef
 
define Package/libip4tc
$(call Package/iptables/Default)
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=IPv4 firewall - shared libiptc library
endef
 
define Package/libip6tc
$(call Package/iptables/Default)
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=IPv6 firewall - shared libiptc library
endef
 
define Package/libxtables
 $(call Package/iptables/Default)
 SECTION:=libs
 CATEGORY:=Libraries
 TITLE:=IPv4/IPv6 firewall - shared xtables library
endef
 
define Package/libipq
  $(call Package/iptables/Default)
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=IPv4/IPv6 firewall - shared libipq library
endef
 
TARGET_CPPFLAGS := \
	-I$(PKG_BUILD_DIR)/include \
	-I$(LINUX_DIR)/user_headers/include \
	$(TARGET_CPPFLAGS)
 
TARGET_CFLAGS += \
	-I$(PKG_BUILD_DIR)/include \
	-I$(LINUX_DIR)/user_headers/include
 
CONFIGURE_ARGS += \
	--enable-shared \
	--enable-devel \
	$(if $(CONFIG_IPV6),--enable-ipv6,--disable-ipv6) \
	--enable-libipq \
	--with-kernel="$(LINUX_DIR)/user_headers" \
	--with-xtlibdir=/usr/lib/iptables \
	--enable-static
 
MAKE_FLAGS := \
	$(TARGET_CONFIGURE_OPTS) \
	COPT_FLAGS="$(TARGET_CFLAGS)" \
	KERNEL_DIR="$(LINUX_DIR)/user_headers/" PREFIX=/usr \
	KBUILD_OUTPUT="$(LINUX_DIR)" \
	BUILTIN_MODULES="$(patsubst ipt_%,%,$(patsubst xt_%,%,$(IPT_BUILTIN) $(IPT_CONNTRACK-m) $(IPT_NAT-m)))"
 
define Build/InstallDev
	$(INSTALL_DIR) $(1)/usr/include
	$(INSTALL_DIR) $(1)/usr/include/iptables
	$(INSTALL_DIR) $(1)/usr/include/net/netfilter
 
	# XXX: iptables header fixup, some headers are not installed by iptables anymore
	$(CP) $(PKG_BUILD_DIR)/include/linux/netfilter/*.h $(1)/usr/include/net/netfilter/
	$(CP) $(PKG_BUILD_DIR)/include/iptables/*.h $(1)/usr/include/iptables/
	$(CP) $(PKG_BUILD_DIR)/include/iptables.h $(1)/usr/include/
	$(CP) $(PKG_BUILD_DIR)/include/libipq/libipq.h $(1)/usr/include/
	$(CP) $(PKG_BUILD_DIR)/include/libipulog $(1)/usr/include/
	$(CP) $(PKG_BUILD_DIR)/include/libiptc $(1)/usr/include/
 
	$(CP) $(PKG_INSTALL_DIR)/usr/include/* $(1)/usr/include/
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libxtables.so* $(1)/usr/lib/
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libip*tc.so* $(1)/usr/lib/
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libipq.so* $(1)/usr/lib/
	$(INSTALL_DIR) $(1)/usr/lib/pkgconfig
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/pkgconfig/xtables.pc $(1)/usr/lib/pkgconfig/
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/pkgconfig/libiptc.pc $(1)/usr/lib/pkgconfig/
endef
 
define Package/iptables/install
	$(INSTALL_DIR) $(1)/usr/sbin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/iptables $(1)/usr/sbin/
	$(LN) iptables $(1)/usr/sbin/iptables-save
	$(LN) iptables $(1)/usr/sbin/iptables-restore
	$(INSTALL_DIR) $(1)/usr/lib/iptables
endef
 
define Package/ip6tables/install
	$(INSTALL_DIR) $(1)/usr/sbin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/ip6tables $(1)/usr/sbin/
	$(LN) ip6tables $(1)/usr/sbin/ip6tables-save
	$(LN) ip6tables $(1)/usr/sbin/ip6tables-restore
	$(INSTALL_DIR) $(1)/usr/lib/iptables
	(cd $(PKG_INSTALL_DIR)/usr/lib/ ; \
		$(CP) libip6t*.so $(1)/usr/lib/iptables/ \
	)
endef
 
define Package/libiptc/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libiptc.so* $(1)/usr/lib/
endef
 
define Package/libip4tc/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libip4tc.so* $(1)/usr/lib/
endef
 
define Package/libip6tc/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libip6tc.so* $(1)/usr/lib/
endef
 
define Package/libxtables/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libxtables.so* $(1)/usr/lib/
endef
 
define Package/libipq/install
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/libipq.so* $(1)/usr/lib/
endef
 
define BuildPlugin
  define Package/$(1)/install
	$(INSTALL_DIR) $$(1)/usr/lib/iptables
	for m in $(patsubst xt_%,ipt_%,$(2)) $(patsubst ipt_%,xt_%,$(2)); do \
		if [ -f $(PKG_INSTALL_DIR)/usr/lib/iptables/lib$$$$$$$${m}.so ]; then \
			$(CP) $(PKG_INSTALL_DIR)/usr/lib/iptables/lib$$$$$$$${m}.so $$(1)/usr/lib/iptables/ ; \
		fi; \
	done
	$(3)
  endef
 
  $$(eval $$(call BuildPackage,$(1)))
endef
 
L7_INSTALL:=\
	$(INSTALL_DIR) $$(1)/etc/l7-protocols; \
	$(CP) files/l7/*.pat $$(1)/etc/l7-protocols/
 
 
$(eval $(call BuildPackage,iptables))
$(eval $(call BuildPlugin,iptables-mod-conntrack-extra,$(IPT_CONNTRACK_EXTRA-m)))
$(eval $(call BuildPlugin,iptables-mod-extra,$(IPT_EXTRA-m)))
$(eval $(call BuildPlugin,iptables-mod-filter,$(IPT_FILTER-m),$(L7_INSTALL)))
$(eval $(call BuildPlugin,iptables-mod-ipopt,$(IPT_IPOPT-m)))
$(eval $(call BuildPlugin,iptables-mod-ipsec,$(IPT_IPSEC-m)))
$(eval $(call BuildPlugin,iptables-mod-ipset,ipt_set ipt_SET))
$(eval $(call BuildPlugin,iptables-mod-nat-extra,$(IPT_NAT_EXTRA-m)))
$(eval $(call BuildPlugin,iptables-mod-iprange,$(IPT_IPRANGE-m)))
$(eval $(call BuildPlugin,iptables-mod-ulog,$(IPT_ULOG-m)))
$(eval $(call BuildPlugin,iptables-mod-hashlimit,$(IPT_HASHLIMIT-m)))
$(eval $(call BuildPlugin,iptables-mod-led,$(IPT_LED-m)))
$(eval $(call BuildPlugin,iptables-mod-tproxy,$(IPT_TPROXY-m)))
$(eval $(call BuildPlugin,iptables-mod-tee,$(IPT_TEE-m)))
$(eval $(call BuildPlugin,iptables-mod-u32,$(IPT_U32-m)))
$(eval $(call BuildPackage,ip6tables))
$(eval $(call BuildPackage,libiptc))
$(eval $(call BuildPackage,libip4tc))
$(eval $(call BuildPackage,libip6tc))
$(eval $(call BuildPackage,libxtables))
$(eval $(call BuildPackage,libipq))
  • Compile it with the whole system or as a single package with:
$ make openwrt/package/iptables/compile V=99

Enabling CHECKSUM in the kernel

  • Enter to the build root directory:
$ cd openwrt
  • Run the kernel menuconfig
$ make kernel_menuconfig
  • Inside the menuconfig check the CHECKSUM option (and dependencies) like follows:
  Symbol: NETFILTER_XT_TARGET_CHECKSUM [=y]
  Type  : tristate
  Prompt: CHECKSUM target support
    Defined at net/netfilter/Kconfig:397
    Depends on: NET [=y] && INET [=y] && NETFILTER [=y] && NETFILTER_XTABLES [=y] && (IP_NF_MANGLE [=y] || IP6_NF_MANGLE [=n]) && NETFILTER_ADVANCED [=y]
    Location:
      -> Networking support (NET [=y])
        -> Networking options
          -> Network packet filtering framework (Netfilter) (NETFILTER [=y])
            -> Core Netfilter Configuration
              -> Netfilter Xtables support (required for ip_tables) (NETFILTER_XTABLES [=y])
  • Exit and compile it with the whole system.

Compiling

There are two ways to compile the package:

With the whole system

Compiling it with the whole system (may take 1-2h).

Compiling as a single package

  • Before that you have to have the whole system previously compiled, at least once.
  • Then, you have to check if all packages that libvirt needs are compiled. If the packages are compiled they must be inside $BUILD_DIR/bin/x86/packages/ as a ipk file.
  • Compile and make the libvirt package.
#Enter $BUILD_DIR directory:
$ cd ~/confine/openwrt/
#Configure, build and create it:
$ make package/feeds/confine/libvirt/compile V=99

Installing and running libvirt

Installing the package

  • Copy the package to the OpenWRT:
$ scp ~/confine/openwrt/bin/x86/packages/libvirt_1.0.2-2_x86.ipk root@$NODE_IP:libvirt_1.0.2-2_x86.ipk
  • You need to update the package list before:
$ opkg update
  • Install libvirt:
$ opkg install libvirt_1.0.2-2_x86.ipk

Note: if shows this error message “Cannot satisfy the following dependencies” and you have made sure they are installed, you can force the package installation with –force-depends option 7).

Running libvirt

After libvirt is installed in the host machine, the daemon must be started up.

For this purpose there’s a init script (section 1.2.1.2) that is placed at the init script directory.

It is executed at the boot process 8), but if it’s not planned to reboot the system it can be started running it manually:

root@OpenWrt:~# /etc/init.d/libvirt-init start
Starting libvirt daemon...
2013-05-07 12:21:22.242+0000: 6369: info : libvirt version: 0.10.2
2013-05-07 12:21:22.242+0000: 6369: warning : virGetHostname:1995 : getaddrinfo failed for 'OpenWrt': Bad value for ai_flags

root@OpenWrt:~# ip addr
[...]
8: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br-local state UP qlen 1000
[...]
95: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether ba:04:ab:1a:00:39 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
107: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
    link/ether ba:04:ab:1a:00:39 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b804:abff:fe1a:39/64 scope link
      valid_lft forever preferred_lft forever

DNSMASQ conflict

Libvirt uses dnsmasq 9) instance to serve DHCP and DNS request from clients on that network.

Then, if you are running your own “global” dnsmasq, this can cause libvirtd to fail to start its dnsmasq and the given virtual network. This happens because both instances of dnsmasq might try to bind to the same port number on the same network interfaces.

If is that case, you may see an error similar to the following:

5006: error : virCommandWait:2271 : internal error Child process (/usr/sbin/dnsmasq
--strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid
--conf-file= --except-interface lo --listen-address 192.168.122.1
--dhcp-range 192.168.122.2,192.168.122.254
--dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases
--dhcp-lease-max=253 --dhcp-no-override) status unexpected: exit status 2

or something like: “Could not start virtual network 'default': internal error”

In order to solve this problem 10) you can do one of these solutions:

1. Disable/stop completely dnsmasq.

2. Edit /etc/dnsmasq.conf and uncomment or add this lines:

interface=eth0

or

 listen-address=192.168.0.1 

(Replace interface or listen-address with the interfaces or addresses you want your global dnsmasq to answer queries on).

And this line to tell dnsmasq to only bind specific interfaces, not try to bind all interfaces:

bind-interfaces

Note: the second solution seems not to work properly in some RD with OpenWRT. If so, dnsmasq can be stopped completely to test if it works and to try the better configuration.

Open Nebula in Confine testbed

Open Nebula is an enterprise-ready open-source platform for sysadmins and devops to manage cloud data centers.

LXC Drivers

Confine network uses LXC 11) as hypervisor on its nodes, however Open Nebula does not support LXC as standard. To be able to use Open Nebula to deploy lxc domains in confine nodes we should use a specific LXC drivers 12).

These drivers have two main features:

  • IM_MAD: a series of remote scripts that are able to monitor the remote hosts
  • VMM_MAD: a series of remote scripts to manage lxc domains.

This driver is very similar with kvm driver because libvirt can support kvm and lxc. They both use virsh command to monitoring hosts and operating virtual machines.

Driver configuration

In order to enable the driver, it is necessary to modify “oned.conf” accordingly. This is achieved by setting the IM_MAD and VM_MAD options as follows:

#-------------------------------------------------------------------------------
# LXC Information Driver Manager Configuration
# -r number of retries when monitoring a host
# -t number of threads, i.e. number of hosts monitored at the same time
#-------------------------------------------------------------------------------
IM_MAD = [
name = "im_lxc",
executable = "one_im_ssh",
arguments = "-r 0 -t 15 lxc" ]

#-------------------------------------------------------------------------------
VM_MAD = [
name = "vmm_lxc",
executable = "one_vmm_exec",
arguments = "-t 15 -r 0 lxc",
default = "vmm_exec/vmm_exec_lxc.conf",
type = "lxc" ]
#-------------------------------------------------------------------------------

The name of the driver needs to be provided at the time of adding a new host to OpenNebula.

Driver files

  • $ONE_LOCATION/etc/vmm_exec/vmm_exec_lxc.conf: Configuration file to define the default values for the LXC domain definitions.
  • $ONE_LOCATION/var/remotes/vmm/lxc/: Scripts used to perform the operations on the lxc domains. These files are called “remotes”, meaning they are copied to the remote hosts and executed there.
  • $ONE_LOCATION/var/remotes/im/lxc.d/: Scripts used to fetch information from the remote hosts (memory, cpu use…). These scripts are copied to the remote hosts and executed there.
  • oned.conf: Example OpenNebula configuration file with the LXC drivers enabled.

Preparing confine nodes for Open Nebula

1. The first step required by Open Nebula hosts is to have a user called oneadmin with password oneadmin as well as a group named oneadmin. Confine image doesn’t have useradd command so we will have to add the user manually. To do it, please login to confine node and follow these steps:

  • Edit /etc/passwd file and add the following line:
oneadmin:x:1001:1001:Oneadmin User:/var/lib/one:/bin/ash

(Make sure that group id is the same as in the frontend.)

  • Edit /etc/groups and add the following line:
oneadmin:x:1001:oneadmin
  • Edit /etc/shadow and add the following line:
oneadmin:password
  • Change oneadmin passwd:
$ psswd oneadmin  -> (new passwd = “oneadmin”)
  • Create home directory for oneadmin user:
$ mkdir /var/lib/one
$ chown oneadmin /var/lib/one
$ chgrp oneadmin /var/lib/one
  • Create .ssh directory
$ mkdir /var/lib/one/.ssh
$ chown oneadmin /var/lib/one/.ssh
$ chmod 700 /var/lib/one/.ssh
  • Create remotes directory
$ mkdir /var/tmp/one
$ chown oneadmin /var/tmp/one

2. The second step is to prepare the passwordless ssh connection between the frontend and each host. We should remember that when we launch Open Nebula in frontend we always do it as oneadmin user. In the frontend we should do the following:

  • Generate ssh key
$ ssh-keygen -t rsa
  • Copy the key to the host.
$ scp ~/.ssh/id_rsa.pub root@hostip:/var/lib/one/.ssh/id_rsa
$ scp ~/.ssh/id_rsa.pub root@hostip:/var/lib/one/.ssh/authorized_keys

Now we should be able to connect from the frontend to the host via ssh without password:

$ ssh oneadmin@hostip

3. Last step is to install swap-utils_2.21.2-1_x86.ipk that we can find avaliable at confine repositorie.

  • Add to: /etc/opkg.conf:
src/gz confine_repo http://repo.confine-project.eu/packages/x86/packages/

New boot & partition system for reliable update in nodes

Up until now, Confine nodes used the following partition scheme:

  • boot (/boot), containing GRUB2 and the kernel
  • root (/rom), containing the system
  • overlay (/overlay), containing the overlayfs for the system
  • home (/home)

Some of its drawbacks were:

  • No reliable update method (no rolling back).
  • Only one confine image may have been installed at a time.

Current update process

Until the proposed scheme below gets implemented, nodes using separate overlay and home partitions will be able to use confine.sysupgrade to upgrade to a newer confine image, keeping these two extra partitions intact. The only drawback of this upgrade script is that it requires that the currently installed and new images be of the exact same size.

The first argument given to the script is the image; It can be compressed in gzip, bz2 or xz, or simply be a raw .img file. The second argument is optional, it specifies the device on which to do the upgrade. It defaults to /dev/sda.

By default, sysupgrade will try and keep the third and fourth partitions (/overlay and /home) and replace the first two with the ones packed in the update image. This can only be accomplished if the sizes of the old and new boot and rootfs partitions are the same, because if they differ (their sizes are sometimes increased) the new partitions will overwrite the overlay partition.

Suppose you have a node with four partitions on disk: “boot”, “root”, “overlay” and “home”. If you'd like to update to a newer image and keep the overlay and home partitions as they are, here's what you should do:

  1. Consider backing up any important data in your overlay and home partitions.
  2. Download a new confine node image, we'll call it CONFINE-IMAGE
  3. Run 'confine.sysupgrade CONFINE-IMAGE [DISK]', where DISK is the target disk to perform the upgrade on. On nodes using hard disks, that would be /dev/sda. On nodes using USB sticks, that would probably be /dev/sdb or /dev/sdc. DISK defaults to /dev/sda.
  4. If it complains about the current and new images differing in size, you can either force the upgrade with -f or start from scratch (removing “overlay” and “home”) with -n. With -f, chances are you will end up with a broken overlay partition. Use it only if you are completely sure that the sizes match.
  5. If it doesn't complain, the script should switch to ramfs, install the new image, recover the third and fourth partitions and reboot. On first boot, it will try and re-enable the overlay and home partitions in fstab. On second boot, you should be brought to the updated system.

Proposed scheme

The new partition and boot scheme would not bring any drawbacks to how the confine images are run, yet it'd bring some advantages. Here is what we have in mind:

  • A sys partition, containing:
    • GRUB2
    • A common initramfs image
    • Each installed node image, consisting of a kernel image and an ext4 rootfs image
    • Symbolic links to the current (and optionally update) node
  • A conf partition, containing:
    • Directories containing configuration files for each node image
    • Directories containing state files for each node image
    • Overlay directories for each node image, to be placed on top of their respective roots
  • A data partition (possibly in another disk), containing sliver template images and their writable overlays.

Notable advantages

  • It will be possible to update to a new confine image, even for testing purposes, without losing the current one. The selected image by default won't be changed, thus even in the case the update doesn't boot, you'll be brought back to the previous system.
  • You'll be able to handle multiple confine systems without needing to fiddle with partitions or boot configurations. The system will provide you with built-in methods to manage your node images.

====== Cloudy Guifi Community Distro ====== Document at BSCW and in this wiki CLOUD SERVICE SOFTWARE.

References

soft/node-software.txt · Last modified: 2015/09/24 15:14 by felix