In this video, I'll close out the topic of Beeline and Impala shell, by showing how you can invoke these two utilities using shell scripts. In previous videos, demonstrating Beeline and Impala shell, I showed how you can run these utilities by issuing Beeline or Impala dash shell commands directly at the operating system command prompt. In this video, I'll show how you can put these commands inside a text file along with other shell commands to create a shell script. Putting shell commands in a script file, makes them easier to rerun later. Instead of entering and running a whole series of commands at the command prompt, or copying and pasting commands from a file you can simply issue a single command to run the whole shell script or you could use a scheduler to run the shell script at some designated times. Even if you're planning to run a series of commands only once, still saving them all in a shell script has some benefits. You can write the commands using an editor instead of right at the command prompt so it's easier to see what you're doing. You'll have a clear record of what commands you ran in case any questions arise. Shell scripts are sometimes called Bash scripts because the most common command shell in the Linux and Unix family of operating systems is called Bash. When you open the terminal on the VM for this course, Bash is the shell you're using there. It's pretty straightforward to include a Beeline or Impala shell command, in a shell script. I'll demonstrate this with an example. Here's a simple shell script named email_results.SH, .SH is the usual file extension for shell scripts. The first line of the script has what's called a hash bin. It tells the operating system to use bash to execute this script. You should generally used that at the top of any shell script. Below that there are two commands. The first command invokes impala shell in a non interactive mode and queries a small subset of the rows from the Flights table. It saved the result to a comma delimited text file named zero_air_time.csv. The second command sends an email to fly at example.com with the specified subject and body and with this file zero_air_time.csv attached to the email. You can write each command on multiple lines in a shell script but you'll need to use a backslash at the end of the line whenever the command continues onto the next line like in this example. After you create a shell script and save it, you need to change the permissions on the script file to allow you to execute it. You can do this using a chmod command like the one shown here. Then to execute the shell script, use the syntax shown here. At the terminal in the current directory where you saved the shell script, Enter./ then the file name of the shell script and press enter to execute it. You can also schedule shell scripts to run at specific times, and you can run shell scripts from inside other scripts or programs or applications. For example you can run a shell script with Python by using the call function in the subprocess module. If you would like to try running this example on the VM, you can create the email_results.SH file that I showed, change the permissions on it and try running it. Just be sure to replace the example email address with your own. I can't guarantee that the mail command will actually get the email to your inbox. It depends on the configuration of your network and on your email provider. When I tested it, it worked for me, but the email ended up in my spam folder. You would have to do a bunch of complicated configuration to get it to work reliably and that's beyond the scope of this video. So as that example demonstrated, you can include Impala shell commands in a shell script simply by entering the commands into the script file in the same way you would enter them at the command prompt and it works the same way for Beeline. Just make sure you invoke Beeline or Impala shell in non-interactive mode in shell scripts. Refer back to the videos in the previous weeks honors lesson to review the arguments that you need to use to do that. Shell scripting is a big topic. I could say much more about it but that's beyond the scope of this course. If you're already familiar with shell scripting, you should have no trouble incorporating Beeline or Impala shell commands into shell scripts. If you want to learn more about shell scripting, there are lots of good online tutorials and books. For example, there are some books from the publisher O'Reilly on this topic. When you include Beeline or Impala shell commands in shell scripts, you'll often want to use other commands to process the result using various Linux utilities. Some of them the more commonly used ones are said OK and grep and there are some books about these two. If you would like to get more practice invoking Beeline or Impala shell and shell scripts, see the reading following this video. In the reading, there's an optional ungraded challenge exercise where you can write a shell script to query the crayons table to return the hexadecimal code for a specific color and then use the results to change the background color on the desktop of the VM. This is meant to be a difficult to exercise and you might need to learn more about shell scripting to complete it.
