Rich0's Gentoo Blog

Gentoo on EC2 From Scratch

with 30 comments

I’ve been beginning to tinker with Amazon EC2 and I figured that many might benefit from a Gentoo-from-scratch recipe. There aren’t too many gotchas. Credit is due to this blog for providing a good non-Gentoo-specific overview.

Before trying any of this, you need to be at least a little familiar with EC2. They have a great getting started walkthrough here. Once you know how to start instances and connect to them you’re halfway there. You’ll also need to set up an S3 account and have an access key and secret key to store your images. On your gentoo box install ec2-ami-tools and ec2-api-tools.

The biggest issue we’re going to have to deal with is that you can’t supply your own kernel. The two issues this creates are udev compatibility and kernel modules. The former is dealt with by setting package.mask, and the latter by hunting down module tarballs online. The modules are fairly optional unless you want to bundle a running image (this requires loop support, which isn’t built in the EC2 kernel).

You might be tempted to start from an existing EC2 AMI. If you find a good one, that isn’t a bad idea, but most of them are VERY out of date. Updating a gentoo install with a pre-EAPI portage and glibc/gcc versions that aren’t even in the tree any longer will be painful. Creating one from scratch isn’t actually that hard.

So, without further ado, here is the recipe (the steps generally follow the Gentoo handbook, so be sure to follow along in that):

  1. Get yourself a stage3 and portage snapshot per the handbook. x86 or amd64 is fine, but be sure to check the Amazon EC2 product page to see which of their offerings are 32/64-bit. This guide is written for 64-bit.
  2. Create yourself a disk image with dd if=/dev/zero of=image.fs bs=1M count=5000 – feel free to tailor the size to your needs but mind the EC2 root filesystem limitations unless you want to run it on elastic storage. The uploaded image will be compressed so you won’t be paying for any unused space in your image.
  3. Point a loopback at your image, format it with a supported filesystem of your choice, and mount it.
  4. Extract the stage3 into your mount. Extract your portage snapshot as well.
  5. Tailor your make.conf as desired – set mirrors/etc. Copy your resolv.conf so that you can chroot.
  6. Mount /proc and /dev per the handbook. If you’re going to install screen bind mount /dev/pts as well.
  7. Chroot into your new environment. Env-update and source /etc/profile. Do an emerge –sync.
  8. Select a profile. Honestly, for most EC2 applications you’re going to want the default 10.0 profile, but it is up to you.
  9. Edit your locales and run locale-gen – sooner or later you’re going to have to update glibc so no sense taking all day to do it.
  10. Set your timezone – I believe EC2 uses GMT but let me know if I’m wrong on that.
  11. Set up your fstab (this is for 64-bit – see the EC2 docs for more details):
    /dev/sda1 / ext3 defaults 1 1
    /dev/sdb /mnt ext3 defaults 0 0
    none /dev/pts devpts gid=5,mode=620 0 0
    none /dev/shm tmpfs defaults 0 0
    none /proc proc defaults 0 0
    none /sys sysfs defaults 0 0
  12. Set net.eth0 to start in the default runlevel
  13. Set a root password. Create and properly set permissions on /root/.ssh/authorized_keys and put anything appropriate in there.
  14. Emerge and set to run syslog-ng.
  15. If you need cron install the cron daemon of your choice. Keep in mind that EC2 instances are volatile so cron is probably less useful than it normally would be.
  16. Add >=sys-fs/udev-125 to your package.mask file.
  17. Populate your world file. A dhcp client like dhcpcd is essential, everything else here is optional:
    app-admin/ec2-ami-tools
    app-admin/ec2-api-tools
    app-admin/syslog-ng
    app-editors/vim
    app-misc/screen
    app-portage/cfg-update
    net-misc/dhcpcd
    sys-process/atop
  18. Do an emerge -au world to install your software, it will downgrade udev and this is a good thing.
  19. Update all your .config files
  20. Remove any unnecessary tarballs, configure user accounts if needed, and do any application setup based upon the purpose of your system.
  21. Add the following to your local.start (credits to whoever created the ami I ripped this out of – I’ll be happy to post them if you identify yourself – as far as I can tell ami owners can’t be identified):

    [ ! -e /root ] && cp -r /etc/skel /root
    if [ ! -d /root/.ssh ] ; then
    mkdir -p /root/.ssh
    chmod 700 /root/.ssh
    fi
    curl http://169.254.169.254/2008-02-01//meta-data/public-keys/0/openssh-key > /tmp/my-key
    if [ $? -eq 0 ] ; then
    cat /tmp/my-key >> /root/.ssh/authorized_keys
    chmod 600 /root/.ssh/authorized_keys
    rm /tmp/my-key
    fi
    killall nash-hotplug
  22. Configure sshd to run at startup, and edit your sshd config to allow root to login.
  23. Exit your chroot, umount anything mounted inside of it.
  24. Clean up tmp, var/tmp, and usr/portage/distfiles, and any other messes you have made. I suspect that to compress the image fully you probably need to zero the free space (dd if=/dev/zero of=file-inside-filesystem BS=1M count=5000 ; rm file-inside-filesystem).
  25. Umount your image and delete any loops you created. Congratulations, you now have a raw image file suitable for EC2. Now we just need to bundle, upload, and register it.
  26. Bundle your image with:
    mkdir out
    ec2-bundle-image --image path-to-image --prefix give-it-a-name --cert path-to-cert-file --privatekey path-to-pk-file --user youramazonaccountnumber --destination out/ --arch x86_64
  27. Upload your image with:
    ec2-upload-bundle --manifest out/prefix.manifest.xml --bucket S3-bucket-name --access-key S3accesskeyhere --secret-key S3secretkeyhere
  28. Register your image with:
    ec2-register S3-bucket-name/prefix.manifest.xml

Congratulations, you now have what should be a working ami. If you ever need to update it you can just chroot into your image, adjust it, and then re-bundle, upload, and register. If you need to delete an ami there is a command that will do it, but I usually just use s3cmd.

You might want to see my updated guide on building with a custom kernel.

Advertisements

Written by rich0

April 2, 2010 at 10:41 am

Posted in gentoo

30 Responses

Subscribe to comments with RSS.

  1. CityCloud do have Gentoo 64bit image, where the initial stages has already been done.

    Trizt

    April 2, 2010 at 3:05 pm

    • Interesting, I didn’t spot it on the list of public ami’s, unless it was the 2008.0 hardened image I tried to use. There are some decent 32-bit gentoo images. Amazon really should provide some way to annotate images, it can be tricky to figure out which ones are worth looking at and of course you pay just to boot them up.

      Do you have a link or other reference to their image?

      rich0

      April 2, 2010 at 3:14 pm

  2. “Point a loopback at your image” — can you elaborate a bit on what this means?

    Vassily Pupkin

    April 3, 2010 at 1:26 pm

    • Your kernel (on the box you’re creating the image from) needs loopback filesystem support enabled (I’m guessing genkernel enables this by default). To use it use the command losetup /dev/loop1 image.fs and now the device /dev/loop1 is pointed at your image. You can do a mkfs.ext3 /dev/loop1 to create a filesystem, or mount /dev/loop1 mountpoint/ to mount it. When you’re done with it and have unmounted the mountpoint use losetup -d /dev/loop1 to unlock the image file and delete the loopback association.

      Loopback filesystems can be very handy things. They basically turn a file into a block device.

      rich0

      April 3, 2010 at 4:25 pm

  3. CityCloud.se offers their own cloud solution, so it’s not Amazon, there will be an international “release” somewhat shortly with English language page at CityCloud.eu.

    There been some talks about switching from Xen to KVM, as Xen seems to give some odd problems in VMs (specially on microsoft VMs).

    Trizt

    April 3, 2010 at 4:03 pm

  4. […] Gentoo on EC2 From Scratch […]

  5. This should actually be integrated in Catalyst. Lots of people have built scripts to make Gentoo AWS images, but Catalyst should be the defacto way to build gentoo for different environments.

    kmind

    April 18, 2010 at 11:59 am

  6. Incredible work, who thought it’d be that easy? It’s quite a powerful setup. I just used your ami-11a14f78 to create a custom lamp server in under an hour. Your /root/.bash_history is also quite helpful.

    Javantea

    April 27, 2010 at 2:22 am

  7. […] This post was mentioned on Twitter by domaci-tutoriali.com. domaci-tutoriali.com said: Gentoo on EC2 From Scratch https://rich0gentoo.wordpress.com/2010/04/02/gentoo-on-ec2-from-scratch/ […]

  8. […] issue with EC2 is that they supply the kernel, and that already caused difficulties with my first EC2 tutorial – the image I created doesn’t let you create a new snapshot from a running image since […]

  9. Does anyone have a guide to create an ebs instance running gentoo. I have tried to convert the existing ami ones to ebs, but the machines never boot

    Nadin

    September 28, 2010 at 10:18 am

    • I haven’t tried it. Did you try using an amazon-provided kernel instead of the custom-built ones? Perhaps they need some config item enabled that I didn’t include.

      I’d certainly be interested in such a guide – maybe if I have time I’ll see if I can get it working.

      rich0

      September 28, 2010 at 12:20 pm

      • I tried using the Amazon ones with no success. The machines would never boot.

        Please let me know if you do figure it out.

        Nadin

        November 23, 2010 at 9:08 pm

  10. A misprint found:

    in

    “On your gentoo box install ec2-ami-tools and ec2-api tools.”

    “ec2-api tools” must be written as “ec2-api-tools”

    taras-lao

    October 25, 2010 at 12:05 pm

  11. Thank you for this howto! How do I create a custom kernel with JFS on /root? I noticed that if I boot into a custom kernel with ext3 on /root I will run out of inodes very fast while still having lots of disk space.

    Any hints?

    Best
    Zeno

    Zeno Davatz

    October 29, 2010 at 4:59 am

    • Well, I don’t know if EC2 supports JFS (I’m guessing it does but you’d have to do your research). If it doesn’t then it won’t be able to load your kernel. Now, if it doesn’t you could put your kernel on an ext3 partition and then mount everything else from another partition of your choosing (since you can put JFS or any other filesystem support in your custom kernel). There are also variants on this using initrds, but no matter what the kernel image itself has to be on a filesystem the EC2 bootloader can read.

      However, there may be an easier fix – when you create an ext3 partition you can specify things like inode space, or allowing room for future volume expansion (useful on LVM/MD/etc). Options like -T small in mkfs.ext3 are probably the easiest way to expand space for inodes. The gentoo handbook recommends this for disks smaller than 4GB.

      Basically mkfs.ext3 scales the number of inodes based on the size of a disk. Any distro is going to have small-file overhead in /etc, /var, etc (Gentoo’s portage tree uses quite a few inodes). This becomes disproportionately large if the OS itself is about the only thing on the disk, which of course is common on EC2. So, you need to allocate space for inodes.

      rich0

      October 29, 2010 at 11:08 am

  12. […] the kernel is a kernel provided by Amazon. Additionally, this guide is based on guides by Rich0 for installing Gentoo on a normal EC2 instance and dkavanagh guide boot from ebs and ami […]

    • Dear Friends,

      I have installed local machine for slackware13.1. And I have created new
      image for 10GB and installed the custom package for slackware13.1

      I have referred the below links and follow-up the steps, for pdf document.

      http://aws.typepad.com/aws/2010/07/use-your-own-kernel-with-amazon-ec2.html

      I am using the kernel is linux-2.6-xenU.x86.tar.bz2. And I have download the kernel below liks
      http://stacklet.com/downloads/kernels/xen/xenU-2.6.31

      I have untar the xenU-2.6.31 kernel and installed the path like (/lib/
      and boot file is stored in /boot path.

      Also I have set the fstab entry for, and changed the above links steps also.

      /dev/sda1 / ext3 defaults 1 1
      none /dev/pts devpts gid=5,mode=620 0 0
      none /dev/shm tmpfs defaults 0 0
      none /proc proc defaults 0 0
      none /sys sysfs defaults 0 0

      And I have created the /boot/grub/menu.lst file and store the file

      default 0
      timeout 3
      title Ec2
      root (hd0)
      kernel /boot/vmlinuz /root=/dev/sda1

      and bundle with amazon aki (aki-407d9529) and upload the amazon server
      and register it,

      While run the instance, I am unable to login the server, I am getting
      below error.

      ec2-get-console-output i-ajht0c9
      i-ajht0c9
      2011-01-29T06:41:40+0000

      Please help me how to fix this issue.

      Ramesh

      Ramesh

      January 31, 2011 at 12:51 am

      • @Ramesh,
        i also expirienced this problem, it’s about a bug on bundle/unbundle script.

        this guru explains it very well:

        https://forums.aws.amazon.com/message.jspa?messageID=189310

        it took me hours, but when you find the root of the problem everything is simple, infact it’s enough emerge at least app-admin/ec2-ami-tools-1.3.57676-r1 (ec2-ami-tools-1.3.34544 is affected by “stdin =” bug)

        @rich0
        thank you! You rocks! Its very useful have a working ami (unfortunatly it’s only us and not eu). Any plans for ebs one?

        m47730

        February 3, 2011 at 7:02 pm

        • Thanks! Been busy lately but I do try to update the images from time to time as well to keep them fresh.

          I’ll definitely try the new ami-tools package – I had tried the latest and greatest at the time without much luck.

          I actually have given thought to ebs – just haven’t gotten around to it. I’m sure something similar will work as long as the right kernel drivers are included and the fstab and kernel are right.

          rich0

          February 3, 2011 at 7:21 pm

        • Oh my god, I’ve been banging my head against this for days!

          Thanks for finding it!

          Jim T

          February 16, 2011 at 5:44 am

  13. @rich0,
    i found a better script to retrieve ssh public key.
    it is more polished (for example check for duplicate keys)… give it a try:

    http://ec2ubuntu.googlecode.com/svn/trunk/etc/init.d/ec2-get-credentials

    m47730

    February 15, 2011 at 10:53 am

    • Thanks. Works great and I updated the images to use it.

      rich0

      February 19, 2011 at 7:28 pm

  14. […] with Gentoo, that you know how layman works in Gentoo, and lastly you know how to gen up an EC2 gentoo image from thin […]

  15. […] be a bad thing. I would hit the IO limit or the CPU limit and actually have to pay money, ugh. From rich0′s blog lets assume your first instance is up and running, has been for a while, and now needs an […]

  16. Are the comments for udev still valid. When I try to downgrade my udev I run into dependency issues with openrc and udev?

    anaken78

    May 28, 2015 at 6:50 pm

    • That is very-much out of date. A lot has changed with udev since then, and with openrc/systemd/etc. This post still contains some useful content, but certainly could use a refresh.

      rich0

      May 28, 2015 at 7:44 pm

  17. For those wanting to switch to systemd, there is a race condition with /etc/local.d/public-keys.start. The easiest work around is while true; wget .., as I changed it to this: https://gist.github.com/Tatsh/0b5da27f0865507d5f5b

    I tried to set the generator to add After=network.target and After=network-online.target. I have dhcpcd.service in multi-user.target.wants. The issue is that dhcpcd never sets the network-online.target to be finished so you end up running the public-keys.start script way before you have a connection.

    A better solution is always welcome.

    You also need to fix or remove the killall-nash script in /etc/local.d. If you never run that service, you can delete the script that kills it on boot.

    Right now based on Pygoscelis-Papua’s image I have a hardened Gentoo with systemd working fully in AWS. Not in degraded state (systemctl status).

    tatsh

    June 22, 2015 at 12:42 pm

    • Obviously this guide predates systemd considerably. One of these days I should update it. I’d probably just create a real service in /etc/systemd/system if possible and then the dependencies can be set correctly.

      rich0

      June 22, 2015 at 1:31 pm

  18. […] are quite a few blogs, such as this, which give a very thorough description of creating Gentoo AMI. However, these blogs primarily tell […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: