path: root/install-directions.txt

INSTALLATION INSTRUCTIONS

(Print using "escript -G" or some other program that handles long lines.)

Note: We've had trouble setting this up on Dell Optiplexes (755 and 760s),
due to difficulties of having framebuffer drivers run on top of Xen.
Since the Xen developers are continuously updating their kernel, and
may eventually merge their stuff into the mainline Linux kernel,
hopefully this will get "silently fixed" over time.


* Backup all necessary files.

* You may need to upgrade your BIOS.  That can be tricky for Dell machines if the BIOS is larger than a floppy (of if you have no floppy).  One simple approach is to create a "DOS" bootable USB stick.  A simple approach is to use the Dell "Diagnostics Utility": http://support.dell.com/support/downloads/download.aspx?c=us&fileid=216318  Here's another approach, which works on some systems but fails on others: You can create a FreeDOS USB stick as described in http://rene.rebe.name/2009-04-19/creating-a-bootable-freedos-usb-stick-thumb-drive by using QEMU to "boot" a CD image, and tell it that the USB stick is the hard drive to install to, as follows on a Linux system (after downloading the FreeDOS ISO CD image): "qemu-system-x86_64 -hda /deb/sdb -cdrom fdbasecd.iso -boot d".  Then copy the BIOS images to the stick, boot the stick, and run the BIOS updater.  Warning: DO NOT upgrade Optiplex GX620 to BIOS version A11, any USB keyboard/mouse will fail to work properly (back off to A10).

* Install Fedora 11 (x86_64):  Take the standard settings, minimal install, EXCEPT for the the drive layout. THIS IS CRITICAL, because the Xen Linux kernel cannot read ext4 filesystems, and Fedora 11's default format is ext4.  At the point where it has you select the drives you will need to tell the system you want to format the drives ext3 rather than ext 4.  This can be accomplished by using the manual review of layout or by using the drop down bar at the top and selecting the custom option.  Consider using separate partitions for /boot, /, and swap, WITHOUT using volumes (e.g., don't use LVM) or the device-mapper; this should work with them, but NOT using them eliminates a potential cause of problems and keeps things simple. Leave lots of extra disk space, so you can use raw partitions for some of the guests (they are faster).  Do NOT press a button in the Fedora 11 installer (anaconda) until the screen redraw is complete (Fedora 11's anaconda is sensitive and can sometimes crash if made to go too fast).  Turn off the "Office and producitivity" group - we don't need those, they will just slow down install.

* DO NOT install the virtualization programs (they'll interfere with our install).

* NOTE: If you have to rebuild/reinstall an OS, and you've previously
  logged into it using ssh, when you try to log into it later the
  client ssh will complain as follows:
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
  This is because installing a Fedora install "from scratch" will
  cause a re-creation of the system RSA host key.  You can save and restore
  that key, if you remember to do so, but it's simple enough to fix the
  client.  If the client in on a Linux system, simply go to $HOME/.ssh
  and edit the file "known_hosts", removing the lines about systems whose
  keys have changed.

* NOTE: If the Fedora 11 installer crashes after the keyboard queries, it may be because it's trying to recover what's on the disk and that is corrupted.  Solve this by booting a rescue disk, and trashing the disk settings, forcing it to start over, e.g.:
     dd if=/dev/zero of=/dev/sda count=1000

* When the installation has you reboot, do so.

* Turn on networking.  One way is to use System/Admin/Network to make sure the connection is managed by the Network Manager, and say that it's "turned on at start".  Then use System/Admin/Services and restart the "Networking" service.  

* Update to current versions.  Log in as root, select Applications/ System Tools/Terminal, and then:
  yum install -y yum-presto  # This speeds up all future yums
  yum update -y

*  Download packages needed to compile Xen (inc. its documents):
  yum groupinstall -y "Development Tools" "Development Libraries"
  yum install -y mercurial dev86 # Mercurial provides "hg" (Xen download)
  yum install -y tetex-latex /usr/bin/fig2dev


* Download xen:
  cd /usr/src
  hg clone http://xenbits.xensource.com/xen-3.4-testing.hg
  cd xen-3.4-testing.hg

* Build. It may seem odd, but build xen and the tools first (Mark Rader learned this): "This is to save some time as the make option asks questions on compiling the kernel and you have to wait by the screen for the rest of the stuff to happen first.  You may need to add packages based upon the installation performed. When you run make, it will download the latest blessed xen kernel.  This is good.  Next it will ask you a series of questions I took the default on most if not all." NOTE: These "builds" will also cause the download of many other files.
  make xen
  make tools
  make

* Install components
  make install-xen install-tools
  make install

* Check /boot to see if you have a new vmlinuz and xen-3.4.gz
  ls /boot

* Crack open the "README" file in the xen-3.4-testing.hg directory and use the directions there to create the initrd file required to boot.  You may need to add aditional drivers such as the framebuffer driver for your particular card.  VESAFB is automatically added (as a built-in, not as a module). As root, I (David) created my initrd this way:
  cd /boot
  depmod 2.6.18.8-xen
  mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod --with=nvidiafb --with=inteldrmfb --with=intelfb initrd-2.6.18.8-xen.img 2.6.18.8-xen

* Make the default inittab level 3 (text) instead of 5 (GUI/X)
  vi /etc/inittab
  ... Change "5" to "3" in initdefault.  The last line should be:
  id:3:initdefault:


  NOTE: You can still log into X as a user, simply run "startx".
  Logging out of X, when running via startx, simply ends X and returns you
  to text I/O.

* Modify /etc/fstab to add /proc/xen (as in the original instructions David sent out), and add xend and xendomains to the list of things to start on boot:

 vi /etc/fstab
 # Add something like:
   xenfs                   /proc/xen               xenfs   defaults        0 0

 chkconfig --add xend
 chkconfig --add xendomains

* Finally you will need to modify /boot/grub/grub.conf to add the xen startup.
  First, set "timeout=" to a non-zero value, say "timeout=5", so you have
  chance to override the default.

  You will need to duplicate the root=UUID=... entry for your system,
  so that it uses the correct drive.
  Basically, we will boot the xen hypervisor /xen-... as the "kernel",
  which will then load 2 modules: the Linux kernel, and the RAMdisk.
  We want it to use vesafb as the video driver, so put that on the "module"
  line that loads the linux kernel.  The vesafb driver will, in turn, load the
  framebuffer driver for your specific graphics system (so the driver for it
  has to be either in the initrd RAM disk or it must be built into the kernel).
  NOTE: If you have /boot as a partition (as recommended here), omit "/boot"
  in the grub.conf file.

  Here's an example:
 title Fedora Xen with VESAFB
        root (hd0,0)
        kernel /xen-3.4.gz video=vesafb:ywrap,mtrr vga=ask
        module /vmlinuz-2.6.18.8-xen ro root=UUID=4542cb11-5a12-4b9c-b4df-2bd2c1decbc3 rhgb quiet
        module /initrd-2.6.18.8-xen.img


  If you're trying to debug the framebuffer driver, you may want to
  add an entry that creates a "stock" Linux kernel and invokes vesafb, e.g.:
 
title Fedora (2.6.29.6-217.2.8.fc11.x86_64) with VESAFB
        root (hd0,0)
        kernel /vmlinuz-2.6.29.6-217.2.8.fc11.x86_64 ro root=UUID=4542cb11-5a12-4b9c-b4df-2bd2c1decbc3 rhgb quiet video=vesafb:ywrap,mtrr vga=ask
        initrd /initrd-2.6.29.6-217.2.8.fc11.x86_64.img


* Install directfb.
  yum install -y directfb directfb-devel
  
  # mark has directfb 1.4.1 compiled from scratch.

* Do this to create xorg.conf.new:
  cd /root
  Xorg -configure

* FOR NOW, disable firewall:
  chkconfig iptables off
  chkconfig ip6tables off

* Login as a normal user.
  Create an ssh key, using ssh-keygen, for whatever user account you want to use.

* Reboot.  Press ESC, and select Xen. (If it works, you can make this default
  by editing /boot/grub/grub.conf)

  It will ask for which framebuffer mode.  Press RETURN, then select.
  I select "j" (1024x768x32), your choices will differ depending on the
  videocard.

* Log in as a normal user, and check out linpicker (copy out to local).
  svn co svn+ssh://secsvn@svn-nitpicker.cl.cam.ac.uk/linpicker

* Change directory into the linpicker material main "trunk"
  cd linpicker/trunk

* Install SCTP library for communication:
  tar xvzf sctplib-*.gz
  cd sctplib-1.0.8
  ./configure --prefix=/usr --libdir=/usr/lib64
  make
  su
  make install
  logout # stop being root
  cd ..

* (Re)Compile linpicker
  make

* If you made changes, check them in:
  svn ci -m "INSERT YOUR MESSAGE HERE"

* Modify /etc/hosts
  and EXPLICITLY ADD the correct IP address, system name, and
  fully-qualified name.  Apache (httpd), if used later, will get confused
  and not work properly without this under some circumstances.

* Set up paravirtual guest.  There are LOTS of ways to do this.
  One technique is suggested by this:
  http://bderzhavets.wordpress.com/2009/06/10/setup-fedora-11-pv-domu-at-xen-3-4-1-dom0-kernel-2-6-30-rc6-tip-on-top-of-fedora-11/

  # Run as root on VM0
  # Start up X, if it's not up already.
  startx

  # Install needed tools
  yum install gitk vnc-server bridge-utils /usr/bin/vncviewer

  vncserver :1 -geometry 1280x1024 -depth 16
  service xend restart
  service xendomains restart

  # Note: Don't "yum install python-virtinst"; this will install
  # old versions of xen-libs.

  # /dev/dvd doesn't show up when in Xen, and that's slow anyway, so
  # when NOT running Xen, do:
  cp /dev/dvd /root/fedora11.iso

  cd /root
  service httpd start
  mkdir -p /var/www/html/f11
  mount -o loop /root/fedora11.iso /var/www/html/f11
  wget http://127.0.0.1/f11/images/pxeboot/vmlinuz
  wget http://127.0.0.1/f11/images/pxeboot/initrd.img

  Put this in /etc/xen/tpath1.startup  (change disk= as appropriate):
  ################
  name="tpath1.startup"
  memory=2048
  disk = ['phy:/dev/sdb3,xvda,w' ]
  vif = [ 'bridge=eth0' ]
  vfb = [ 'type=vnc,vncunused=1']
  kernel = "/root/vmlinuz"
  ramdisk = "/root/initrd.img"
  vcpus=2
  on_reboot = 'restart'
  on_crash = 'restart'
  ################


  # Now run:
  xm create tpath1.startup
  vncviewer localhost:0


  # Now install as usual.  For "installation media" select "URL".
  # You'll need to set up the IP configuration.
  # Give the URL as http://MACHINE_NAME/f11
  # This will install onto the given disk as specified above.
  # I recommend having separate /boot, /, swap, and NOT using LVM.
  # I give 2000M for /boot, 2000M for swap, and the rest to /.
  # DO NOT use ext4; use ext3.

  # Uncheck "Productivity tools", we don't need them on the guest.
  # I suggest NOT installing the "development tools" at this time.
  # There are so many updates that it's faster to install initially
  # without them, and then use "yum groupinstall..." to install them.
  # It will try to reboot; on VM0, try to "xm shutdown NUMBER", but
  # that will probably fail, so then use "xm list"; "xm destroy NUMBER"
  # At this point, you may want to "dd" or "cp" the partition to safe place.
  # If you use "cp" you can force sparse copying, which saves space, e.g.:
  #    cp --sparse=always /dev/sdb3 /root/tpath1-cp2.img

  
  Now, create a normal boot, by (on VM0) putting this in /etc/xen/tpath1.para
  (change the disk= as appropriate):
  ################
  name="tpath1.para"
  memory=2048
  disk = ['phy:/dev/sdb3,xvda,w' ]
  vif = [ 'bridge=eth0' ]
  vfb = [ 'type=vnc,vncunused=1']
  bootloader = "/usr/bin/pygrub"
  vcpus=2
  on_reboot = 'restart'
  on_crash = 'restart'
  ################

  # Now run (on VM0, with X running):
  xm create tpath1.para && vncviewer localhost:0
  # Complete the initial install.

  You may need to reconfigure the network, esp. since
  the MAC address will have changed.  Go to network manager,
  edit it, and under "Hardware device" turn off "Bind to MAC address",
  and enable the network connection.
  You may want to turn off being controlled by NetworkManager, and
  instead, select System/Administration/Services and
  turn on the "service" of "network".

  Now you can set up the paravirtual guest. E.G., on the guest, type:
  yum install -y yum-presto
  yum update -y

  Copy linpicker files over; recommend "rsync -a" to do this. E.G.:
  rsync -a linpicker/ tpath1:/home/dwheeler/linpicker/


  # On the guest, set it up so that X can find the screen driver:
  ln -s /home/dwheeler/linpicker/trunk/XClient/screen/ovlscreen_drv.so.1.0.1 /usr/lib64/xorg/modules/drivers/ovlscreen.so

  Modify guest's selinux to disable for now, or cp the driver there
  and re-run SELinux file attributes.
  (Otherwise, X won't follow the symlink to the new driver.) E.G.:
  vi /etc/selinux/config

  Similarly, on the guest, disable the filewalls (as root):
  chkconfig iptables off
  chkconfig ip6tables off

* If you want to use the GUEST as a development environment, you will
  need to set it up.  We haven't done that, but here are some notes
  on how to do this.  Basically,  you need to repeat some of the
  steps above to get the development files on the guest.  For example:

  # You'll want to do this on the guest to easily install sctplib:
  yum groupinstall -y "Development Tools" "Development Libraries"

  yum install -y dev86 tetex-latex /usr/bin/fig2dev
  yum install -y directfb directfb-devel
 
  Repeat the above steps for setting its /etc/inittab to default to 3,
  getting the linpicker source,
  "Install SCTP library for communication".
   Install directfb.

  You may want to add accounts using "useradd" and "passwd".

* Pick a system  you're going to do development on, and go on.

* Install X development packages
  yum install -y xorg-x11-server-devel

  # You need this to compile "track", b/c it uses a normal user-level interface
  yum install -y libX11-devel

* Compile
  cd ~/linpicker/trunk
  make
  cd XClient/screen
  make

* From VM0 you can control the guests:
  xm list            # Show list of guests
  xm shutdown NUMBER # Cleanly shutdown guest NUMBER
  xm destroy NUMBER  # Forceably shutdown guest NUMBER
  xm create NAME     # Create VM

  To start up a client guest, do something like this:
  xm create tpath1.para
  Then use xm list to find the domain ID of the new guest.
  Edit the file /etc/linpicker/domaininfo/GUEST_IP_ADDRESS
  to contain this ID number (replacing the name GUEST_IP_ADDRESS
  with the actual decimal IP address).


* After compiling, if your development system != a guest, rsync the
  compiled files to the guest.

* On the guest, configure as root:

  mkdir -p /etc/linpicker
  echo "IP-ADDRESS-OF-CLIENT" >  /etc/linpicker/hosts
  echo "IP-ADDRESS-OF-SERVER" >> /etc/linpicker/hosts

  Once the guest is set up, you may want to shut it down
  (you can do this cleanly as root using "poweroff now")
  and make a backup of its partition using "dd".


* Start up.  First, on VM0, as root start up the linpicker server:
  cd ~dwheeler/linpicker/trunk
  ./linpicker-server

  Control-C to kill it.

* Start up a Linpicker client.  Log into a VMx (say, via ssh), as root, and do:
  cd ~dwheeler/linpicker/trunk
  X -config ./client.xorg.conf   # This invokes the screen driver

  control-\ will kill it this way.
  it writes to the log file .debug-commo.log


* To use doxygen to generate some documentation. Before doing so:
  - Make sure there is a "docs" directory under the trunk.
    mkdir -p ~/linpicker/trunk/docs
  - Install required packages:
    yum install graphviz doxygen

  Change directory to be in the "trunk", and run doxygen:
    doxygen Doxyfile

  This will generate detailed source file documentation.


NOTES

* You can edit the X.org configuration, and run it using:
  Xorg -config xorg.conf.new

* Note: mkinitrd doesn't work properly on Xen.
  We haven't currently tracked down the cause, because there are 2 easy
  workarounds.  One is to reboot to a standard kernel, of course.
  The other is to uncompress the initrd ramdisk, copy stuff in, and
  recompress.  You can do this as follows:

  mkdir -p /root/temp
  cd /root/temp
  # Extract cpio contents into this dir:
  zcat /boot/initrd-2.6.30-rc6-tip.img 2>&1 | cpio -idv | less
  # Copy v86d in:
  cp -p /sbin/v86d sbin/v86d
  # Create new cpio archive, with this addition:
  find . -depth -print | cpio -ov | \
     gzip -c > /boot/initrd-2.6.30-rc6-tip.img.NEW



* Setup/download source code of linpicker/nitpicker.

* Setup firewall.  You must allow SCTP communication between client
  and server.  Eventually, we intend to set the rules so that specific
  IP addresses can only come from specific locations (normally IP addresses
  cannot be trusted, but since we can force all connections to be
  point-to-point through virtual connections, in this case we can).

* Build (make)

* Run linpicker.  You need to start the server on VM0, and the
  guest on a different VM.  When it runs, you should see the DirectFB
  splash screen as a start.


Note: The Fedora 11 instructions here:
http://bderzhavets.wordpress.com/2009/06/10/setup-fedora-11-pv-domu-at-xen-3-4-1-dom0-kernel-2-6-30-rc6-tip-on-top-of-fedora-11/
Do not work with framebuffers.  We tried to make it work with uvesafb and v86d, to no avail.


* To install IPP printers, you may need to remove a package on Fedora 11:
   yum remove cups-pk-helper
  For this bug and work-around, see: 
    https://bugzilla.redhat.com/show_bug.cgi?id=447266



* Note: You can install python-virtinst, but this will install old xen tools:

yum install python-virtinst
Loaded plugins: presto, refresh-packagekit
Setting up Install Process
Dependencies Resolved
================================================================================
 Package                   Arch       Version                 Repository   Size
================================================================================
Installing:
 python-virtinst           noarch     0.400.3-8.fc11          fedora      401 k
Installing for dependencies:
 iscsi-initiator-utils     x86_64     6.2.0.870-8.fc11        fedora      750 k
 libvirt                   x86_64     0.6.2-14.fc11           updates     1.9 M
 libvirt-python            x86_64     0.6.2-14.fc11           updates     117 k
 qemu-img                  x86_64     2:0.10.6-1.fc11         updates     101 k
 xen-libs                  x86_64     3.3.1-11.fc11           fedora      176 k
...
(1/6): iscsi-initiator-utils-6.2.0.870-8.fc11.x86_64.rpm | 750 kB     00:03     
(2/6): libvirt-0.6.2-14.fc11.x86_64.rpm                  | 1.9 MB     00:11     
(3/6): libvirt-python-0.6.2-14.fc11.x86_64.rpm           | 117 kB     00:00     
(4/6): python-virtinst-0.400.3-8.fc11.noarch.rpm         | 401 kB     00:02     
(5/6): qemu-img-0.10.6-1.fc11.x86_64.rpm                 | 101 kB     00:00     
(6/6): xen-libs-3.3.1-11.fc11.x86_64.rpm                 | 176 kB     00:00     
--------------------------------------------------------------------------------
Total                                           107 kB/s | 3.4 MB     00:32 

You can eliminate the "wrong" xen libraries this way:
 yum erase xen-libs


SOURCE FILES

An easy way and thorough way to determine which source files are used
by which program is do an strace on its recompile (this detects
all indirect uses), e.g.:

 rm linpicker-server # Or whatever executable
 rm *.o
 strace -e trace=open -v make linpicker-server > ,1 2> ,2
 grep '\.[ch]\"' ,2 | grep -v / > source-files-linpicker-server
 # Edit with vi or sed to remove surrounding open("...")
 # Use "sort -u" to eliminate duplicates

Source files used for linpicker-server:
buffer.c
client.c
client.h
comm_interp.c
comm_interp.h
comm-server.c
comm-server.h
comm-setup.c
comm-setup.h
comm-structs.h
externs.h
fbif.h
linpicker-server.c
print-data.c
print-data.h
sctp.h
sctp_wrapper.c
sctp_wrapper.h
typedefs.h
view.c
view.h

Source files used for multiplex:
comm-client.c
comm-client.h
comm-keyboard.c
comm-keyboard.h
comm-mouse.c
comm-mouse.h
comm-setup.c
comm-setup.h
comm-structs.h
lxevent.h
multiplex.c
sctp.h