Spring 2016 - Projects On Big Data Software (I590 - Geoffrey C. Fox)¶

Overview¶

Projects on Big Data Software course introduces lessons in two sections: Theory and Technology. Units in Technology are listed in this page. To navigate Theory Units, syllabus and discussions, please use the course site scholargrid.org.

Schedule for Units in Technology Section

Notice¶

System Notice

Gaining Access to FutureSystems and Core Technologies¶

In this unit, you will learn how to gain access to the FutureSystems resources. It includes the portal account creation, class project participation, SSH key generation and login node access. Some of other lessons have been prepared for the beginners to help understand the basics of Linux operating systems and the collaboration tools i.e. GitHub, Google Hangout and Remote Desktop. Please watch video lessons and read through web contents.

Guidelines¶

• Assignments must be completed individually.
• Discussion is allowed (e.g. via Slack) but the submission should be made by yourself. Acknowledge your helpers/collaborators name in the submission if you discussed or got help from anyone.
• Use an individual github repository. A repository in FutureSystems will be given later.

Tasks¶

Complete the following tasks. Place all answers in a file named HW2-$USERID.txt and submit it via IU Canvas. (Replace $USERID with your email name at IU e.g. HW2-albert.txt if your email address is albert@indiana.edu)

Example view of your submission:

1. albert
2. ...
3. ...
9. http://...

Guidelines¶

• Assignments must be completed individually.
• Discussion is allowed (e.g. via Slack) but the submission should be made by yourself. Acknowledge your helpers/collaborators name in the submission if you discussed or got help from anyone.
• Use an individual github repository. A repository in FutureSystems will be given later.

Create IU GitHub Account¶

• Simply login https://github.iu.edu with your IU Username and Password (It is a same IU Credential that you use on other IU sites e.g. one.iu.edu)

OpenStack Command Line Tool nova¶

OpenStack Kilo is ready to use (as of 02/04/2016) on FutureSystems and you will have a virtual instance (server) using OpenStack Command Line Tool nova, if you complete all the tasks in this assignment. The tasks you need to complete are:

• SSH into india.futuresystems.org and

  • enable nova command

• Register a SSH key on OpenStack

  • rsa type

  • with default key file names

    • public: $HOME/.ssh/id_rsa.pub
    • private: $HOME/.ssh/id_rsa

  • passphrase enabled

• Start a single instance:

  • on fg491 project
  • with a m1.small flavor,
  • a Ubuntu-15.10-64 image,
  • the registered key above,
  • and hw3-$OS_USERNAME vm name
  • Assign a Floating IP address

• Install required software on a virtual instance

  • virtualenv
  • pip
  • ansible

Test Program¶

We provide hw3.py test file in your repository, checkout hw3 branch. Run this on india.futuresystems.org, if you completed all tasks above. All available tests should be succeeded without errors. First, clone your private repository from IU GitHub. See details here: IU GitHub Guidelines. You will use virtualenv to prepare packages.

Run:

bash setup.sh
source $HOME/bdossp_sp16/bin/activate

Now, you can run the test program:

python hw3.py

Completed all? You may see:

...........
----------------------------------------------------------------------
Ran 11 tests in 1.646s

OK

Find hw3-results.txt file after you ran hw3.py python program in your current directory. Add this file in your IU GitHub repository.

FAQ¶

Q. Where should I run the test program hw3.py? A. It is on india.futuresystems.org, not your VM instance.

Q. bash setup.sh produces command not found errors. A. Make sure you can use nova command to start a new VM like you did in the hw3 tasks. Otherwise, the test program can’t verify what you accomplished.

Q. The hw3.py test program was failed due to missing python package named lib. A. Run hw3.py in a main directory of the hw3 branch. hw3.py itself won’t work. Helper functions are required.

Submission via IU GitHub (github.iu.edu)¶

From now on, you will use IU GitHub to submit assignments on a private repository. IU GitHub Guidelines

1. Clone your private repository from the course organization. You IU Username is the name of your repository.
2. Create a hw3 branch

git branch hw3
git checkout hw3

3. Run pull command to fetch and merge with the template repository:

   git pull git@github.iu.edu:bdossp-sp16/assignments.git hw3

4. Sync with remote:

   git push -u origin hw3

5. Add hw3-results.txt to your repository:

   git add hw3-results.txt

6. Merge the template

   git commit -am "initial merge with the template"
   

7. Sync your changes:

   git push -u origin hw3

Challenging Tasks (Optional)¶

The following tasks are optional but strongly recommended to try. These are related to Python packages and APIs (application program interface). OpenStack nova is also extended to get more experience.

* Use virtualenv named 'bdossp-sp16' in your home directory
* Open a web port to the Flask application to allow access from outside

1. albert
2. ...
3. ...
9. http://...

Useful links¶

• Python lesson: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/linux/python.html
• OpenStack Beginners: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/iaas/openstack.html
• OpenStack QuickGuide: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/quickstart_openstack.html
• OpenStack Operations Guide: http://docs.openstack.org/openstack-ops/content/user_facing_operations.html

Guidelines¶

• Assignments must be completed individually.
• Discussion is allowed (e.g. via Slack) but the submission should be made by yourself. Acknowledge your helpers/collaborators name in the submission if you discussed or got help from anyone.
• Use an individual github repository. A repository in FutureSystems will be given later.

Use hw5 branch¶

• Login https://github.iu.edu with your IU Username and Password (It is a same IU Credential that you use on other IU sites e.g. one.iu.edu)
• Checkout hw5 branch

MongoDB Ansible Role¶

Writing Mongodb playbook is taught in the Ansible lessons. You write MongoDB Ansible role in this assignment. Submit inventory, main playbook, command script and your role including sub-directories.

Requirements¶

The following files should be included in your submission

• inventory file
• site.yml the main playbook file
• mongodb directory (which is ansible role for mongodb)
• hw5-cmd.script file

Preparation¶

• Login to india.futuresystems.org

• Use the same bdossp-sp16 virtualenv used in hw3

• Install ansible to the bodssp-sp16 virtualenv via python package manager

• Change a directory to your IU GitHub repository where you work on hw5

• Create a new branch hw5 by:

  git checkout -b hw5
  

• Pull hw5 template files by:

  git pull git@github.iu.edu:bdossp-sp16/assignments.git hw5
  

• Sync to remote by:

  git push origin hw5
  

• Start working on hw5

HW5 Tasks¶

You need to write an Ansible role to install mongodb on your vm instance hw3-$USER. Ansible Playbook for MongoDB installation is given in the Ansible lessons. You may start from there but you need to install MongoDB on Ubuntu 15.10 at this time. Systemd is a main init system in Ubuntu 15.10 and you need to locate a service file using Ansible modules. Certain conditions should be met in your submission, see the requirements below:

• create a new Ansible role where

  • mongodb is a role name

• Describe tasks in tasks directory

  • Add the MongoDB public GPG key from:

    • hkp://keyserver.ubuntu.com:80
    • Use EA312927 as MongoDB public GPG Key ID when Ubuntu package management imports a key (apt-key)

  • Install mongodb-org with 3.2 Community Edition for Ubuntu Trusty 14.04 LTS by adding a MongoDB repository from:

    • deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse

• Define those as Ansible variables in defaults directory, at least the four variable names below should be used:

  • mongodb_keyserver (to store hkp://...)
  • mongodb_gpgkey_id (to store EA312...)
  • mongodb_repository_list (to store deb http://...)
  • monogodb_package_name (use ‘mongodb’)
  • (more vars can be defined)

• Two handlers

  • one for starting mongodb
  • one for restarting mongodb

• Locate a service file where:

  • destination is /lib/systemd/system/mongodb.service
  • owner/group of the destination file is root
  • mode of the file is 0644
  • reload mongodb after adding this file to remote
  • You can find mongodb.service.j2 template file in your hw5 branch

• Write a main playbook:

  • to include your new role
  • in site.yml file

• Run hw5.sh to record your outputs in hw5-cmd.script file

Submission via IU GitHub (github.iu.edu)¶

Use IU GitHub to submit assignments on a private repository. IU GitHub Guidelines

1. Clone your private repository from the course organization. You IU Username is the name of your repository.
2. Create a hw5 branch

git branch hw5
git checkout hw5

3. Run pull command to fetch and merge with the template repository:

   git pull git@github.iu.edu:bdossp-sp16/assignments.git hw5

4. Sync with remote:

   git push -u origin hw5

5. Add files and directories to your repository:

   git add inventory git add mongodb git add site.yml git add hw5-cmd.script

6. commit

   git commit -am "submission hw5"
   

7. Sync your changes:

   git push -u origin hw5

Challenging Tasks (Optional)¶

The following tasks are optional but strongly recommended to try. These are to write mongodb roles for RedHat-based operating system as well using Ansible conditionals and different modules, if necessary.

Possible Project idea (Running Ansible on Windows)¶

Develop Ansible Playbooks and Roles for Windows machines using PowerShell and winrm Python package instead of SSH. You may find multiple ways like:

• develop a PowereShell script that starts a VirtualBox and runs the Debian ansible in it, have a local key be used see the instalation instructions of Cloudmesh that let you set up ssh on a windows machine also.
• develop a Docker based ansible container. However this is not as straight forward as the key management need to be done right.

You can find more information here Windows Support

Useful links¶

• Ansible Basic: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/ansible.html
• Ansible Playbook: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/ansible_playbook.html
• Ansible Role: http://bdossp-spring2016.readthedocs.org/en/latest/lesson/ansible_roles.html
• Ansible Best Practices: https://docs.ansible.com/ansible/playbooks_best_practices.html
• Ansible official documentation: http://docs.ansible.com/ansible/index.html

News¶

• NIST Fingerprint Example (03/09/2016)
• HBase is now supported (03/02/2016)
• Examples are under development (03/02/2016)
• Projects, datasets, and technologies from the past are available (02/26/2016)

Important Dates¶

• Project Proposal: March 18th
• Oral Presentation: Week 12 - April 1st, 2nd (Tentative)
• Progress Checkup: Week 14 - April 15th, 16th (Tentative)
• Final Submission: April 29th

Submission¶

• IU GitHub: https://github.iu.edu/bdossp-sp16

Team Coordination¶

Up to 3 members is recommended but individual is allowed.

Project Expectation (Grade)¶

Final project counts as 60% of semester grade and 40% goes on assignments.

• 60% Final project

  • 10% Proposal
  • 10% Presentation
  • 30% Source code
  • 10% Report

Project Style¶

• Basic
• Bonus

You do not require strong background or programming skills with HPC or Hadoop to complete a final project. We’ve noticed that, however, there are some difficulties learning Linux systems, shells, or scripts and improving programming skills with parallelization in general. You have two options, Basic and Bonus, to start your project based on your capability on these.

Project Choice¶

• Deployment
• Benchmark (Performance Test)
• Parallelization
• Analytics
• Created Own (upon approval)

Project Requirement¶

• Installation/Configuration by Ansible playbook or relevant tools (Ansible Roles)
• Reproducibility - runnable on Linux distribution
• Sample Dataset - up to 480GB per team
• 12 VM instances with m1.medium are given to the utmost each team
• Software Stacks similar to Software Layers

Project Copyright¶

Your project deliverables may be introduced in the future classes or be shared by others online after the end of semester.

Project Proposal¶

Please submit your project proposal by due. The proposal.rst RST file is provided in the project template repository. Fork this repository and write your proposal in the file under ‘docs’ directory. Find RST Quick Reference , Online RST Editor here. A project proposal is typically 1-2 pages long and should contain in the description section:

• the nature of the project and its context
• the technologies used
• any proprietary issues
• specific aims you intent to complete
• and a list of intended deliverables (atrifacts produced)

• Project Proposal Example
• Project Template Repository

Oral Presentation¶

You are required to demonstrate your project during the presentation week. The clear statement of problems are necessary with schedule, plan, role of team members, resources to use.

• A student will use Adobe Connect to give a presentation which will be recorded.
• 3-5 minutes per team.
• Presentation can be substituted with written reports upon approval. 1-2 page progress report(s) need to be included.

Progress Checkup¶

The following activities will be evaluated:

• Code development in a project repository
• Participation of team members
• Software installation
• Datasets preparation

List of Possible Projects¶

• Big Data Analytics Stack

  • Deployment project using Ansible Playbooks (Ansible Roles)

You may consider to work on Big Data Analytics Stack using Ansible Playbooks. The default configuration of the stack is YARN + HDFS + Java + Hadoop MapReduce, Spark, and Pig. You can develop a new addon for one of the optional software and attach to your stack. Find more details here big-data-stack, Ansible Roles.

List of Datasets¶

• Examples from the previous class

  • List of Datasets 2015 Fall

List of Technologies¶

• ABDS and HPC Technologies and Software Stacks

• Examples from the previous class

  • List of Technologies 2015 Fall
  • List of Technologies (2015 Spring)

FAQ¶

17. Use of FutureSytem is required?

A. No, it is not required, but it must be deployable and runnable on FutureSystems Kilo and you should provide detailed instructions on how to do so. Ideally, running ansbile-playbook site.yml should be all that is needed to deploy, after booting and editing inventory.txt file.

17. I need more time to complete code development, may I have an extension?

A. Extension would be approved upon request. Send an extension request email message to the course email with a title [Project Extension] and an expected completion date.

Q. Our team wants to change a topic or scope of a project after project proposal or presentation, is it allowed?

A. Topic should be close to what you proposed earlier. Please contact Dr. Fox or Course Email if you change a topic or a scope of your project significantly. Also inform if you change team members. These changes would be approved upon request.

17. Report or survey type of final project is allowed?

1. No, software project is only allowed in this class.

Q. I found there is a similar project that I proposed, should I keep working on my project?

A. Consult with Course Team to make differences in detail. You may be asked to focus on specific area in order to avoid similarity.

17. Can’t make a oral presentation because I have a business trip (or a conference).

1. Schedule a meeting in Week 11 or Week 13 with Course Team.

17. What does that mean there is no direct support on Datasets and Analytics software?

1. We will provide support for accessing a dataset under 500 GB

Questions & Support¶

• Course Email: bdosspcoursehelp@googlegroups.com
• Google Hangout (voice & screen share): upon request

Useful Links¶

• Analytics tools

  • Spark Quick Start Tutorial with Python
  • Pig Getting Started
  • Flink Python API
  • MLlib
  • Storm Tutorial
  • Ibis Tutorial

• Scheduler

  • Running Marathon with Apache Mesos

• Database

  • Getting Started with MongoDB
  • M101P: MongoDB for Developers (MongoDB Online Courses
  • HBase Getting Started
  • Cassandra

• GUI Tools

  • MongoDB Web Interfaces
  • MySQL phpMyAdmin
  • MySQL Workbench
  • HeidiSQL for MySQL Windows

• Python Tools

  • Pydoop
  • IPython Notebook (Jupyter)

• Visualization

  • Highcharts Getting Started
  • Pyplot Tutorial

How to ask a question¶

When submitting a questions please:

1. copy/paste directly from the terminal into the email message. Do not send text files, zipfiles, or other attachments as they may be not be opened. Screen grabs may be acceptable if you annotate the relevant parts using an image editor, or they are integral to showing the problem.

2. use the mailing list (bdosspcoursehelp@googlegroups.com) to direct questions as all support staff are subscribed. Direct message via Slack is discouraged.

3. describe the specific step you are trying to accomplish, include the actions you took and the results. Ensure that you can reproduce your problem by executing only the steps present in your email. Also include:

   • your futuresystems username
   • the output of nova show MY_VM (replacing MY_VM appropriately)

   For example:

   I cannot ssh into my VM due to permission denied errors:
   
   $ ssh user@machine
   Permission denied (publickey,hostbased).
   
   I tried following the steps in the FAQ as defined here: https://...
   <SHOW THE COMMANDS AND THEIR OUTPUT>
   
   
   Relvant information is:
   
   Username: albert
   $ nova show MY_VM
   +--------------------------------------+----------------------------------------------------------+
   | Property                             | Value                                                    |
   +--------------------------------------+----------------------------------------------------------+
   | accessIPv4                           |                                                          |
   | accessIPv6                           |                                                          |
   | config_drive                         |                                                          |
   | created                              | 2016-03-29T19:35:41Z                                     |
   | fg491-net network                    | 10.0.5.22, 149.165.159.241                               |
   | flavor                               | m1.large (4)                                             |
   | hostId                               | 683eed6c03fcc23879620b6042eaaa22149d915bfcd4ec9e5feab7c5 |
   | id                                   | 60dc4420-e5c0-4897-9a58-2cd48d6521b0                     |
   | image                                | CentOS7 (dc5c041f-7881-441e-af91-e9620efde901)           |
   | key_name                             | india                                                    |
   | metadata                             | {}                                                       |
   | name                                 | $USER-myvmname                                           |
   | os-extended-volumes:volumes_attached | []                                                       |
   | progress                             | 0                                                        |
   | security_groups                      | default                                                  |
   | status                               | ACTIVE                                                   |
   | tenant_id                            | 74e411d6d99e4497901d4c4e2b159f41                         |
   | updated                              | 2016-03-29T19:35:56Z                                     |
   | user_id                              | 090ea72e85c94c49ad8cc8133627fa1a                         |
   +--------------------------------------+----------------------------------------------------------+
   

Why can’t I ssh into my VM?¶

1. Make sure to boot using on the internal network first. Once the node is up, then attach the floating ip.

   $ NET_ID=$(nova net-list | grep $OS_PROJECT_NAME | cut -d' ' -f2)
   $ nova boot --nic net-id=$NET_ID      # ... etc
   $ nova floating-ip-list | grep ' - '  # find an available floating ip.
   $ # create a floating ip with nova floating-ip-create if there are no floating ips available
   $ nova floating-ip-associate          # ... etc
   

2. If your VM is on the 10.1.x.y it is accessible:

   • from outside the subnet with a floating ip only
   • from inside the 10.1.x.y subnet

   Check ip of your VM(s) with

   $ nova show $USER-myvmname | grep network
   

3. Make sure you can ping your VM:

   $ ping -c 5 $IP
   

4. Make sure that the machine have finished boot by checking that ssh daemon is listening on port 22:

   $ nc -zv $IP 22
   Connection to 149.165.158.1 22 port [tcp/ssh] succeeded!
   

5. Make sure you are trying to log in with the correct username. For Ubuntu VMs, the username is ubuntu; for CentOS use centos.

   For example, to log into an Ubuntu VM:

   $ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@$IP
   

6. If you are still unable to ssh, try a hard reboot a few times:

   $ nova reboot --hard $USER-myvmname
   

7. Check that you have an ssh key registered with openstack using nova keypair-list and make note of the fingerprint:

   $ nova keypair list
   +----------------+-------------------------------------------------+
   | Name           | Fingerprint                                     |
   +----------------+-------------------------------------------------+
   | india          | 41:29:20:a2:51:25:5d:66:71:02:15:b6:cd:e2:36:06 |
   +----------------+-------------------------------------------------+
   

8. Check that the correct key name was passed to nova boot when starting the VM by using nova show:

   $ nova show $USER-myvmname
   +--------------------------------------+----------------------+
   | Property                             | Value                |
   +--------------------------------------+----------------------+
   # ...
   | key_name                             | india                |
   # ...
   +--------------------------------------+----------------------+
   

9. Ensure that the fingerprint matches:

   $ ssh-keygen -lf ~/.ssh/id_rsa
   2048 41:29:20:a2:51:25:5d:66:71:02:15:b6:cd:e2:36:06 ~/.ssh/id_rsa.pub
   

10. Make sure that the key was injected into the VM during the startup by grabbing the console log and searching for your fingerprint. Make sure to wait for a few minutes after nova boot to allow the node start up:

$ nova console-log $USER-myvmname | grep -A 2 -B 4 '41:29:20:a2:51:25:5d:66:71:02:15:b6:cd:e2:36:06'
ci-info: ++++++Authorized keys from /home/centos/.ssh/authorized_keys for user centos+++++++
ci-info: +---------+-------------------------------------------------+---------+-----------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |  Comment  |
ci-info: +---------+-------------------------------------------------+---------+-----------+
ci-info: | ssh-rsa | 41:29:20:a2:51:25:5d:66:71:02:15:b6:cd:e2:36:06 |    -    |           |
ci-info: +---------+-------------------------------------------------+---------+-----------+

If, after going through these steps, you are still unable to access the VM, delete the VM and try again two or three times, waiting a few minutes between each attempt. OpenStack is a collection of many distributed systems, and the nature of distributed systems is that they can be prone to random failure.

If you are still unable to log in, please contact us and indicate that you have gone through these steps, and show the output of the above commands.

Why can’t I modify my ~/.ssh/authorized_keys file?¶

You can not manually manage your authorized_keys file on india for security reasons. If you need to change your ssh key, do so via the SSH keys tab on your Web Portal Account.

Why does my MongoDB deployment fail?¶

In this case: mongodb is installed successfully, but the service cannot be started. Solving this is the goal of the assignment, which is demonstrating an important aspect of many development processes: namely the affects of changing infrastructure.

To put this in context: Ubuntu for many years (through the 14.04 LTS release) used the Upstart init daemon. As of 15.04, this is switched to systemd. However, the mongodb installation expects to use Upstart to run the service, which therefore fails.

There are many solutions to this type of problem:

1. add the system service file by hand
2. rollback the OS from Ubuntu 15.04 to 14.04
3. use a different repository which includes the systemd service file

For the purposes of this homework, the first option is taken, and the service file is provided in the repository. As the hw instructions say place the provided service file in the appropriate location.

If, after deploying the service file you are still unable to start the mongodb service, please include the contents of /lib/systemd/system/mongodb.service in your email.

One common issue is the user the mongodb service runs as: you should make sure that the username in the service file matches the user account created for mongodb.

• Check the username in the service file by looking at the User value.
• Check the username on the system by grep -i mongo /etc/passwd

If these two values do not match, adjust your ansible deployment.

Security Groups¶

As projects are shared and everyone can modify the security groups, it is best to create security groups prefixed the your username: eg $USER-default and add your rules to that.

Naming VMs¶

All VMs should be prefixed by your username. This will allow everyone to identify the VMs that belong to your.

Naming the key on the Cloud¶

It is best to name the key on the cloud with your <portalname> in order not to confuse that with others its also good practice to optionally put -key behind it, SO your key name would be <portalname>-key.

Accessing Root¶

The default login user (ubuntu on India, cc on Chameleon) has sudo privileges.

Beware of Denial-of-Service attacking your own machine¶

We have seen students looping through an ssh command and as soon as it failed issued a new one during boot. They did it so many times that they flodded the network as they did it not just with one but many many vms. Multiply this by X users and you can see that alone through this process you can create a denial of service attack on cloud services. So what you have to do is to put a sleep between such ssh attemts to see if your vm is really up. Put at least 30 seconds At time it can be as much as 10 minutes dependent on usage.

You can do this in bash and zsh using until:

$ until nc -zv $IP 22; do sleep 30s; done

This sleeps for seconds each iteration until the ssh service is detect to be available on port 22.

Using Chameleon Cloud¶

You can find documentation on how to migrate from India (Futuresystems) OpenStack to Chameleon cloud here: https://github.com/futuresystems/class-admin-tools/blob/master/chameleon/big-data-stack.org

Make sure you follow these instructions.

Regarding some common questions about switching to and using Chameleon, here are some tips if you are having trouble:

$ nova list
ERROR (DiscoveryFailure): Cannot use v2 authentication with domain scope

$ nova list
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
... terminating nova client

env | egrep '^OS_' | cut -d= -f1 | while read var; do unset $var; done