The first thing we need to do is to modify our redshift cluster iam role to allow write to s3. We go to our cluster in the redshift panel, we click on properties, and then we will see the link to the iam role attached to the cluster. we click on it and it will open the IAM role page.

Then we add the policy named AmazonS3ReadOnlyAccess as shown in the following pic:

And with this we already have all the required permissions ready. The next step now is to make sure we have an available s3 bucket. I’ve created one for the demo purposes with a folder called parquet_exports.

For starting the extraction, we will use the customers table we used in the previous articles. This table was loaded as well from the TPC-DS test data from s3 in a gzip file but now it sits inside our redshift node. The instruction to unload the data is called UNLOAD

Since we want our data in parquet + snappy format, which is usually the recommended way (avro is not supported in redshift UNLOAD, only CSV and parquet), we need to express it in the unload statement.

Contrary to spectrum, here we can unload data to buckets in another region. So if thats your need make sure you fill the bucket region as well in the unload statement. The statement is as follows

UNLOAD ('select-statement')
TO 's3://object-path/name-prefix'
authorization
[ option [ ... ] ]

where option is
{ [ FORMAT [ AS ] ] CSV | PARQUET
| PARTITION BY ( column_name [, ... ] ) [ INCLUDE ]
| MANIFEST [ VERBOSE ] 
| HEADER           
| DELIMITER [ AS ] 'delimiter-char' 
| FIXEDWIDTH [ AS ] 'fixedwidth-spec'   
| ENCRYPTED [ AUTO ]
| BZIP2  
| GZIP 
| ZSTD
| ADDQUOTES 
| NULL [ AS ] 'null-string'
| ESCAPE
| ALLOWOVERWRITE
| PARALLEL [ { ON | TRUE } | { OFF | FALSE } ]
| MAXFILESIZE [AS] max-size [ MB | GB ] 
| REGION [AS] 'aws-region' }

As you can see, it allows to pass a select statement instead of a table name. This way we can project the required columns and there is no need to export the whole table if we do not need it. We can also specify the max filesize. Usually with parquet 256 MB is a good split size. And make sure with Parquet not to specify any compression format as otherwise it will crash, as by default redshift already compresses it with Snappy.

To run the export we also need to fetch the arn of our redshift role, and incliude it just after the bucket path:

UNLOAD ('select * from customer')
TO 's3://albertnogues-parquet/parquet_exports/'
iam_role 'arn:aws:iam::742123541312:role/Redshift_Albertnogues.com'
FORMAT AS PARQUET
MAXFILESIZE 256 MB

After about two minutes, the query finished sucessfully. We can go to our s3 bucket to see the parquet files there and check that the split file size is the one we requested

We can check first the size of the table in redshift with the following query:

SELECT "table", tbl_rows, size size_in_MB FROM SVV_TABLE_INFO
order by 1

So its quite clear that the export looks ok as the size is similar. We can now download one fo the parquet files and inspect it with some parquet tool analyzer. I tend to use the python version of parquet-tools based on apache arrow project. You can install it with:

pip install parquet-tools

And then we will inspect the file with the following:

parquet-tools inspect 0001_part_03.parquet

And if we scroll down a little bit we can see the total number of files on our parquet file:

The post Unload data from AWS Redshift to S3 in Parquet appeared first on Albert Nogués.

There are a few requirements however to be able to use redshift spectrum. First, the data loaded will be like an external table in other systems (RDBMS) or like hive external tables. So this means that we can forget about updating or deleting our data. We will only get read only capability, but usually thats we are looking for (think in historical data from already closed exercices, archived data and so on), where the cost of storing it in s3 even in a cool tier can ve several orders of magnitude cheaper than keeping it inside redshift.

There are other technical limitations tough. For example, the s3 bucket has to be in the same region as our redshift cluster, so plan in advance or move the data to another s3 bucket. There are other requirements like permissions and so on, you can read it here.

Before starting we need to attach a role to our redshift cluster that grants access to read s3 buckets. If you followed my previous article on redshift you will already have this role, otherwise check how to do it here.

If you followed my previous article, apart of the s3 read only permisison, you need to add the glue catalog permission to create a table. For this, modify the role (or if you’re creating a new one, add the following permission: AWSGlueConsoleFullAccess

For the sake of this exercise I’ve load part of the TPC-DS customer data into a s3 bucket i made public on the Paris region. Then we will create an external customer table pointing to the data in the bucket (which is in fact compressed in gz) and we will query that data from redshift.

Once the bucket data is loaded, we are ready to go back to our redshift cluster. I’ve used this file for the tests. We will create an external schema and an external customers table. For this we need first to copy the arn of the role we created with the s3 read access. To get that let’s go to our cluster, then click on properties and later on click in Copy Amazon Resource Name (ARN):

Once we have the arn of our role, then run the folloowing in your editor. It will create the external schema in a new spectrum database (feel free to choose any other database name if you need)

create external schema spectrum_albertnogues 
from data catalog 
database 'spectrum' 
iam_role 'arn:aws:iam::742123541312:role/Redshift_Albertnogues.com'
create external database if not exists;

Then we are ready to create our external table:

create external table spectrum_albertnogues.customer
(
  c_customer_sk int4,                 
  c_customer_id char(16),             
  c_current_cdemo_sk int4 ,   
  c_current_hdemo_sk int4 ,   
  c_current_addr_sk int4 ,    
  c_first_shipto_date_sk int4 ,                 
  c_first_sales_date_sk int4 ,
  c_salutation char(10) ,     
  c_first_name char(20) ,     
  c_last_name char(30) ,      
  c_preferred_cust_flag char(1) ,               
  c_birth_day int4 ,          
  c_birth_month int4 ,        
  c_birth_year int4 ,         
  c_birth_country varchar(20) ,                 
  c_login char(13) ,          
  c_email_address char(50) ,  
  c_last_review_date_sk int4
)
row format delimited
fields terminated by '|'
stored as textfile
location 's3://redshift-spectrum-albertnogues/customers/';

As you can see it’s not really 100% the same customers table from the TPC-DS data. This is because external tables do not support primary keys, not null syntax and some other keywords that do not make sense in external tables. Make sure you used the right separator and selected the right format of data (compression is determined by the file extension).

Then, as you will see the table creation takes virtually nothing. This is because in fact, the data is not loaded, its only a shortcut to our s3 data. But to make sure it’s working we can run a few queries to our new table:

select count(*) from spectrum_albertnogues.customer;

And to make sure data is properly loaded, we can query it as well:

select * from spectrum_albertnogues.customer LIMIT 5;

As a last example, if you run my previous blog post on redshift, we can now join our new external customers table with our web returns fact table, to see how many objects these users returned depending on their origin country, to see if there is any pattern or not.

select c_birth_country as customer_birth_country, sum(wr_return_quantity) as qty_returned_total
 from web_returns,
      date_dim,
     spectrum_albertnogues.customer
 where wr_returned_date_sk = d_date_sk 
   and d_year =2002
   and wr_returning_customer_sk = c_customer_sk
 group by c_birth_country
 order by 2 desc;

Of course the query should take a bit more since the data is not inside redshift, and needs to be fetched from s3 but we can get our output quite fast and easily:

Thats it!. For more advanced topics like performance improvements when querying from s3 and others, you can read here and here. I recommend basically partitioning, a splittable file format like parquet compressed with snappy or some other codec, small files (but not very small), add as many filers as you can to avoid retrieving unnecesary data, and if you use a columnar format like parquet, you will avoid fetching unused columns in your select statement. This is important (and can have a huge impact in costs) because now it’s the time of discussing the price of all this.

Bear in mind that AWS charges 5 dollars for each Terabyte of data scanned from s3 with spectrum, so make sure you do the right usage on this (only data infrequently queried) and do not overestimate these charges as they grew quickly.

For expert advice or project requests you can contact me here. Happy Querying!

The post Use Redshift Spectrum to query infrequently used data on S3 appeared first on Albert Nogués.

I’ve split the article in 4 steps that cover diverse topics:

• Part 1. Deploying a redshift cluster.
• Part 2. Load TPC-DS data.
• Part 3. Run queries to verify the performance.
• Part 4. Performance tricks.

1. Deploying a redshift cluster

As always with cloud providers, we have many ways to deploy resources. We can use the native tools (aws cli) or the portal, or we can use some external software, like terraform.

For this example we will go through the aws portal as we only plan to create one for test purposes. If you never created one, there is a promotion on aws that let’s you to create a two month dc2.Large node for free.

To create the cluster go to the redshift area of the aws portal here and click on create cluster. For this we will need to define the name of the cluster, the size and the type of cluster (production, free test), the number of nodes, the name of the database, admin username and password, the port and other aspects like vpc and subnet, security groups, parameter group and backup policies we want to apply. We can leave the defaults for these. For the rest of values choose non used ones and a strong user and password combination. If you want to use the free ofer, make sure you select the same as me:

The fill the rest of the information and press the create cluster button. In a few minutes the cluster will be up and running.

2. Load TPC-DS Data

Before starting loading the data, let’s connect to our brand new cluster. For connecting, we can use the integrated editor in the aws portal or use some other software like dbeaver. I personally recommend the latter as it’s more flexible. If our usual client does not support redshift we can always use the postgre driver.

Then, DBeaver will automatically download the jar driver from internet unless we already have it and we then have to configure our connection:

But before connecting, lets open the port in the security group. For this we need to go to our cluster in the aws portal, click on properties:

And check for the Network and Security section. If we want to access the cluster from our computer, as it’s the case (although in a corporate environment probably you dont want this!) we need to enable the publickly accessible section. This change will take some time.

But before being able to connect we still have to click in the VPC security group and add a rule to be able to access redshift port from our location:

With this we can now test the connection from DBeaver, and now the connection should be succesful:

So, now, the next step is to load the information into redshift. For this, we will load some tables from a public s3 bucket maintained by aws. We will create four tables and then populate them from data in a public s3 bucket. We can get the ddl from here. We will crate the following three tables:

create table customer_address
(
 ca_address_sk int4 not null ,
  ca_address_id char(16) not null ,
  ca_street_number char(10) ,      
  ca_street_name varchar(60) ,   
  ca_street_type char(15) ,     
  ca_suite_number char(10) ,    
  ca_city varchar(60) ,         
  ca_county varchar(30) ,       
  ca_state char(2) ,            
  ca_zip char(10) ,             
  ca_country varchar(20) ,      
  ca_gmt_offset numeric(5,2) ,  
  ca_location_type char(20)     
  ,primary key (ca_address_sk)
) distkey(ca_address_sk);

create table customer
(
  c_customer_sk int4 not null ,                 
  c_customer_id char(16) not null ,             
  c_current_cdemo_sk int4 ,   
  c_current_hdemo_sk int4 ,   
  c_current_addr_sk int4 ,    
  c_first_shipto_date_sk int4 ,                 
  c_first_sales_date_sk int4 ,
  c_salutation char(10) ,     
  c_first_name char(20) ,     
  c_last_name char(30) ,      
  c_preferred_cust_flag char(1) ,               
  c_birth_day int4 ,          
  c_birth_month int4 ,        
  c_birth_year int4 ,         
  c_birth_country varchar(20) ,                 
  c_login char(13) ,          
  c_email_address char(50) ,  
  c_last_review_date_sk int4 ,
  primary key (c_customer_sk)
) distkey(c_customer_sk);

create table date_dim
(
    d_date_sk                 integer               not null,
    d_date_id                 char(16)              not null,
    d_date                    date,
    d_month_seq               integer                       ,
    d_week_seq                integer                       ,
    d_quarter_seq             integer                       ,
    d_year                    integer                       ,
    d_dow                     integer                       ,
    d_moy                     integer                       ,
    d_dom                     integer                       ,
    d_qoy                     integer                       ,
    d_fy_year                 integer                       ,
    d_fy_quarter_seq          integer                       ,
    d_fy_week_seq             integer                       ,
    d_day_name                char(9)                       ,
    d_quarter_name            char(6)                       ,
    d_holiday                 char(1)                       ,
    d_weekend                 char(1)                       ,
    d_following_holiday       char(1)                       ,
    d_first_dom               integer                       ,
    d_last_dom                integer                       ,
    d_same_day_ly             integer                       ,
    d_same_day_lq             integer                       ,
    d_current_day             char(1)                       ,
    d_current_week            char(1)                       ,
    d_current_month           char(1)                       ,
    d_current_quarter         char(1)                       ,
    d_current_year            char(1)                       ,
    primary key (d_date_sk)
) diststyle all;

create table web_returns
(
wr_returned_date_sk int4 ,   
  wr_returned_time_sk int4 , 
  wr_item_sk int4 not null , 
  wr_refunded_customer_sk int4 ,
  wr_refunded_cdemo_sk int4 ,   
  wr_refunded_hdemo_sk int4 ,   
  wr_refunded_addr_sk int4 ,    
  wr_returning_customer_sk int4 ,
  wr_returning_cdemo_sk int4 ,   
  wr_returning_hdemo_sk int4 ,  
  wr_returning_addr_sk int4 ,   
  wr_web_page_sk int4 ,         
  wr_reason_sk int4 ,           
  wr_order_number int8 not null,
  wr_return_quantity int4 ,     
  wr_return_amt numeric(7,2) ,  
  wr_return_tax numeric(7,2) ,  
  wr_return_amt_inc_tax numeric(7,2) ,
  wr_fee numeric(7,2) ,         
  wr_return_ship_cost numeric(7,2) ,
  wr_refunded_cash numeric(7,2) ,   
  wr_reversed_charge numeric(7,2) ,  
  wr_account_credit numeric(7,2) ,   
  wr_net_loss numeric(7,2)           
  ,primary key (wr_item_sk, wr_order_number)
) distkey(wr_order_number) sortkey(wr_returned_date_sk);

You can create a new schema or use the public one for this test. I’ve created all four in the public schema.

Once the tables have been created, then lets populate them. To populate the four tables we need to pass some AWSKeys or a Role with permission to read data from S3 (even if the bucket is public). If it’s a one off job maybe the keys can be quick to set up, but if we intend to do this often, we should create an IAM role and attach it to the Redshift cluster. We will go this second way. For this lets go to the IAM portal and create a new role following this instructions.

Then in the Redshift section, go to Actions, Manage IAM Roles, find your new IAM role and add it to the redshift cluster

After a few seconds / minutes you should see the new role being added to the cluster:

Then we are ready to move to DBeaver and start the data loading. Since we have to load three tables we will use the following commands (First click on copy Amazon Resource Name (ARN) from the role we just added to our cluster, and then ran the three following commands (leave the region as us-east-1 no matter from where you are launching it as the data we are loading it’s in a bucket in that region. And replace my ARN by yours:

copy customer_address from 's3://redshift-downloads/TPC-DS/2.13/3TB/customer_address/' credentials 'aws_iam_role=arn:aws:iam::742123541312:role/Redshift_Albertnogues.com' gzip delimiter '|' EMPTYASNULL region 'us-east-1';

copy customer from 's3://redshift-downloads/TPC-DS/2.13/3TB/customer/' credentials 'aws_iam_role=arn:aws:iam::742123541312:role/Redshift_Albertnogues.com' gzip delimiter '|' EMPTYASNULL region 'us-east-1';

copy date_dim from 's3://redshift-downloads/TPC-DS/2.13/3TB/date_dim/' credentials 'aws_iam_role=arn:aws:iam::742123541312:role/Redshift_Albertnogues.com' gzip delimiter '|' EMPTYASNULL region 'us-east-1';

copy web_returns from 's3://redshift-downloads/TPC-DS/2.13/3TB/web_returns/' credentials 'aws_iam_role=arn:aws:iam::742123541312:role/Redshift_Albertnogues.com' gzip delimiter '|' EMPTYASNULL region 'us-east-1';

The load of the data should finish sucesfully after several minutes. For the biggest table (web_returns) it took me 50 minutes. After the load we can do a few checks (counts):

select 'customer_address' table_name, count (*) from customer_address
UNION
select 'customer' table_name, count (*) from customer
UNION
select 'date_dim' table_name, count (*) from date_dim
UNION
select 'web_returns' table_name, count (*) from web_returns

select count(*) from customer_address;  -- 15000000
select count(*) from customer;  -- 30000000
select count(*) from date_dim;  -- 73049
select count(*) from web_returns;  -- 216003761

3. Query the data

We can now launch a query to see the results:

-- start template query30.tpl query 3 in stream 1
with /* TPC-DS query30.tpl 0.75 */ customer_total_return as
 (select wr_returning_customer_sk as ctr_customer_sk
        ,ca_state as ctr_state, 
 	sum(wr_return_amt) as ctr_total_return
 from web_returns
     ,date_dim
     ,customer_address
 where wr_returned_date_sk = d_date_sk 
   and d_year =2002
   and wr_returning_addr_sk = ca_address_sk 
 group by wr_returning_customer_sk
         ,ca_state)
  select  c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag
       ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address
       ,c_last_review_date_sk,ctr_total_return
 from customer_total_return ctr1
     ,customer_address
     ,customer
 where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
 			  from customer_total_return ctr2 
                  	  where ctr1.ctr_state = ctr2.ctr_state)
       and ca_address_sk = c_current_addr_sk
       and ca_state = 'OH'
       and ctr1.ctr_customer_sk = c_customer_sk
 order by c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag
                  ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address
                  ,c_last_review_date_sk,ctr_total_return
limit 100;

And the query returns the first 100 results in just 4 minutes:

4. Performance Tricks

With Redshift as almost all databases, there is always room for improvement. The possibilities for tuning start with compression and then go to distribution and sort keys. Right now, after loading the 4 tables, these are taking (we can see in storage used in the main page of our cluster in the aws redshift portal) :

11.88% (19.01 of 160 GB used)

Which is not bad taking into account that in .gz the size of the files in total is about 15.37 GB, so the db has done pretty much a good job (Caution, while loading you need much more space than this, as the bulk load needs stagging space before being able to optimize it, based on my tests you need 3 times the final space at least) . But this can be improved. We didnt specify any compression when we defined our table structure. The rationale here is that if we compress the data, the engine wil process less volume of data and the resoults should come faster right?. Let’s see:

analyze compression customer;

As we can see there are optimizations to be made to save both space (and possibly) processing time. Let’s create a better table:

create table customer_compressed
(
  c_customer_sk int4 encode delta not null ,                 
  c_customer_id char(16) encode zstd not null ,             
  c_current_cdemo_sk int4 encode az64,   
  c_current_hdemo_sk int4 encode az64,   
  c_current_addr_sk int4 encode az64,    
  c_first_shipto_date_sk int4 encode az64,                 
  c_first_sales_date_sk int4 encode az64  ,
  c_salutation char(10) encode zstd,     
  c_first_name char(20) encode zstd,     
  c_last_name char(30) encode zstd,      
  c_preferred_cust_flag char(1)  encode zstd,               
  c_birth_day int4 encode az64,          
  c_birth_month int4 encode az64,        
  c_birth_year int4 encode az64,         
  c_birth_country varchar(20) encode bytedict,                 
  c_login char(13) encode lzo,          
  c_email_address char(50) encode zstd,  
  c_last_review_date_sk int4 encode az64,
  primary key (c_customer_sk)
) distkey(c_customer_sk);

And then insert back:

insert into customer_compressed select * from customer;

And then later:

SELECT "table", tbl_rows, size size_in_MB FROM SVV_TABLE_INFO
order by 1

So we easily saved 25% of space in this table. Let’s do the same for the rest (mainly for web_returns, the others are small so probably no need to compress them)

analyze compression web_returns;

but unfortunately we see that there is no improvement to be made.

So the compression is over. Next steps are the distribution and the sort keys but being tpc-ds queries these have been already optimized for the tpc-ds queries. For this you can follow the excellent guide in aws documentation here.

The post Load data from s3 and run TPC-DS queries on amazon Redshift. appeared first on Albert Nogués.

If you want to read more about it you can read Microsoft post on their website.

Basically there are two versions, version 1, and version 2. You can see the differences between both in this post. I recommend using version 2 when possible as it’s more performant. As started on Microsoft website:

The primary difference and reasons for updating the Windows Subsystem for Linux from WSL 1 to WSL 2 are to:

• increase file system performance,
• support full system call compatibility.

WSL 2 uses the latest and greatest in virtualization technology to run a Linux kernel inside of a lightweight utility virtual machine (VM). However, WSL 2 is not a traditional VM experience.

So, let’s see the steps to make all this work. First of all we need to enable wsl in windows optional features. For this we can go through the control Panel or the windows search and make a few clicks to enable it or we can run some command from powershell. Make sure you run them as administrator (Right click to Powershell, run as Admin…)

The commands are either one of these two:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

or

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Any of the two will do the work. After it, restart your computer.

(OPTIONAL) If optionally you want to upgrade to wsl v2, you need to make sure you are running one of the newer windows 10 releases:

• For x64 systems: Version 1903 or higher, with Build 18362 or higher.
• For ARM64 systems: Version 2004 or higher, with Build 19041 or higher.
• Builds lower than 18362 do not support WSL 2. Use the Windows Update Assistant to update your version of Windows.

If this is the case, you can simply then upgrade it to wsl v2 by running the following to enavle the Virtual Machine Platform support:

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Download and install the v2 kernel package from here.

And set wsl v2 as the default mode with the following command:

wsl --set-default-version 2

Then we are good, we can open now the microsoft store and download any of the supported images there, like Ubuntu, Suse, Debian, Fedora, Alpine… But if you want to use Centos8, we need some extra work.

There is a Github project that now has been archived due to the Centos / Centos Stream controversy but is still working for our purpose. We can use it as long as we are in Windows 10 version 2004, otherwise upgrade your windows 10.

We go to the release page of the github repo and download the zip file. Another option is to go to the new github repo continuing the work of the original one with the new Centos versions (including CentOs stream and CentOs 8.3 here), the procedure will be the same.

Once downloaded, we extract the zip file and we run the exe file inside. This will unpack all the content of the root filesystem inside the tar.gz file and will make everything ready to be run.

To enter into the bash shell we simply click again into the exe file once it has been installed and voilà!. Another way is to write the bash command in the cmd, which will open bash inside our cmd window.

Then we can update our system to the latest version by running dnf update:

[root@DESKTOP-JONAOMM ~]# cat /etc/redhat-release
CentOS Stream release 8
[root@DESKTOP-JONAOMM ~]# dnf update
CentOS Stream 8 - AppStream                                                             3.2 MB/s | 6.4 MB     00:02
CentOS Stream 8 - BaseOS                                                                2.5 MB/s | 2.4 MB     00:00
CentOS Stream 8 - Extras                                                                 10 kB/s | 7.0 kB     00:00
Dependencies resolved.
========================================================================================================================
 Package                                    Arch       Version                                      Repository     Size
========================================================================================================================
Installing:
 kernel                                     x86_64     4.18.0-259.el8                               baseos        5.0 M                     baseos         31 M
...
Installing weak dependencies:
 dconf                                      x86_64     0.28.0-4.el8                                 appstream     108 k

Transaction Summary
========================================================================================================================
Install   53 Packages
Upgrade  138 Packages

Total download size: 326 M
Is this ok [y/N]:

At the same time I recommend you to install windows terminal as well to get better experience (tab support, search, custom themes…)

If we look around we can see our local volumes “mounted” inside

We can also share our image with others (friends, coworkers) if we customized it or installed some software which is great for people like me that have many friends, which I met in sites like https://chatempanada.com online.

. We can export it with:

wsl --export ImageName FileName

with Image Name the name of the linux flavour we want to export. We can check all the available ones with the following command:

wsl --list
Distribuciones del subsistema de Windows para Linux:
CentOS8 (predet.)
CentOS8-stream

And then pack the tar file inside a compressed file and send it to whom we want.

Another useful trick is that if we dont want to log in as root (which is what wsl by default does), we can run the following:

wsl -d ImageName -u username

Happy Linuxing!

The post HowTo run (Any Linux) Centos8 inside windows 10 appeared first on Albert Nogués.

There are a few prerequisites before starting with the installation that need to be configured or otherwise the installation can/will crash. To address some of these I wrote some small scripts in ansible to automatize the adjustment of these parameters or the data copy. The scripts are not finished, and I plan to improve them adding a few parts that I still haven’t automatized like the creation of the yum repositories for installing cloudera manager and a few other small tasks.

For this manual we assume that Cloudera manager has already been installed, and the database to hold Cloudera manager data repository and some of the other tools has been installed as well.

The script is written in ansible and it has three playbooks and an inventory file. The inventory file is in yaml format. I grouped the nodes between master(s) / workers (datanodes) the Cloudera manager server and the gateway. The inventory file is defined as follows and you can save it as inventory.yaml:

all:
  children:
    cm:
      hosts:
        cm_host:
    master:
      hosts:
        master_host:
    workers:
      hosts:
        worker1_host:
        worker2_host:
        worker3_host:
    gateway:
      hosts:
        gateway_host:

Replace the xxx_host by the fully qualified domain name of your server.

Then we have the prerequisites playbook, this one has more substance. Apart from the prerequisites enunciated in Cloudera’s website I added some tweaks and actions like the change of the mysql jdbc driver, as the one in yum is outdated and will make the creation of the dbs to crash in the wizard. You can save this one as cloudera_prerequisites.yaml:

---
- hosts: all
  connection: ssh
  remote_user: youruser
  become: yes
  become_method: sudo
  become_user: root
  tasks:
   - service: name=firewalld state=stopped enabled=False
   - selinux: state=disabled
   - sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present
   - sysctl: name=net.ipv6.conf.default.disable_ipv6 value=1 state=present
   - sysctl: name=vm.swappiness value=1 state=present
   - shell: sysctl -w vm.swappiness=1
   - copy: src=/etc/hosts dest=/etc/hosts owner=root group=root mode=0644
   - yum: name=java-1.8.0-openjdk-devel state=latest
   - systemd: name=tuned state=started
   - shell: tuned-adm off
   - systemd: name=tuned state=stopped enabled=False
   - name: Disable THP support scripts added to rc.local
     lineinfile:
       path: /etc/rc.local
       line: |
         echo never > /sys/kernel/mm/transparent_hugepage/enabled
         echo never > /sys/kernel/mm/transparent_hugepage/defrag
   - name: Change permissions of /etc/rc.local to make it run on boot
     shell: chmod +x /etc/rc.d/rc.local
     become_method: sudo
   - service: name=ntpd state=started
   - name: Allow 'simigsolutions' user to have passwordless sudo
     lineinfile:
       path: /etc/sudoers
       state: present
       regexp: '^youruser'
       line: 'youruser ALL =(ALL) NOPASSWD: ALL'
#   - name: Install mysql jdbc driver
#     yum:
#       name: mysql-connector-java
#       state: latest
   - name: download newer jdbc for mysql to avoid crash
     get_url: url=https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz dest=/tmp/mysql-connector-java-5.1.46.tar.gz
   - name: Check if /tmp/mysql-connector-java-5.1.46.tar.gz exists
     stat:
       path: /tmp/mysql-connector-java-5.1.46.tar.gz
     register: stat_result
   - block:
     - name: Extract downloaded jdbc
       unarchive:
         src: /tmp/mysql-connector-java-5.1.46.tar.gz 
         dest: /tmp/
     - name: Creates directory for the java driver if it does not exist
       file:
         path: /usr/share/java
         state: directory
         mode: 0755
         recurse: yes
     - name: Copies the file
       copy:
         src: /tmp/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar
         dest: /usr/share/java/mysql-connector-java.jar
         owner: root
         group: root
         mode: 0644
     - name: Copies the file to sqoop as well
       copy:
         src: /tmp/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar
         dest: /var/lib/sqoop/mysql-connector-java.jar
         owner: sqoop
         group: sqoop
         mode: 0644		
     when: stat_result.stat.exists == True

The script can be called with the following: ansible-playbook -i inventory.yaml cloudera_prerequisites.yaml –ask-pass –ask-become-pass

I’ve also created two aditional playbooks, one to create folders in the mount points to store hdfs and yarn data:

---
- hosts: workers
  connection: ssh
  remote_user: youruser
  become: yes
  become_method: sudo
  become_user: root
  tasks:
  - name: Creates directory datanodes
    file:
      path: /home/data/dfs/dn
      state: directory
      owner: hdfs
      group: hdfs
      mode: 0700
      recurse: yes
  - name: Creates directory namenodes
    file:
      path: /home/data/yarn/nm
      state: directory
      owner: yarn
      group: yarn
      mode: 0700
      recurse: yes
  - name: Creates directory namenodes
    file:
      path: home/data/impala/impalad
      state: directory
      owner: impala
      group: impala
      mode: 0700
      recurse: yes

- hosts: master
  connection: ssh
  remote_user: youruser
  become: yes
  become_method: sudo
  become_user: root
  tasks:
  - name: Creates directory namenode
    file:
      path: /home/data/dfs/nn
      state: directory
      owner: hdfs
      group: hdfs
      mode: 0700
      recurse: yes
  - name: Creates directory secondary namenode
    file:
      path: /home/data/dfs/snn
      state: directory
      owner: hdfs
      group: hdfs
      mode: 0700
      recurse: yes

And another one to copy some jdbc driver and distribute it in all machines of the cluster (sql server) but can be adapted to any downloadable jdbc driver:

- hosts: all
  connection: ssh
  remote_user: youruser
  become: yes
  become_method: sudo
  become_user: root
  tasks:
   - name: download sqlserver jdbc driver
     get_url: url=https://download.microsoft.com/download/4/D/C/4DCD85FA-0041-4D2E-8DD9-833C1873978C/sqljdbc_7.2.2.0_enu.tar.gz dest=/tmp/sqljdbc_7.2.2.0_enu.tar.gz
   - name: Check if /tmp/sqljdbc_7.2.2.0_enu.tar.gz exists
     stat:
       path: /tmp/sqljdbc_7.2.2.0_enu.tar.gz
     register: stat_result
   - block:
     - name: Extract downloaded jdbc
       unarchive:
         src: /tmp/sqljdbc_7.2.2.0_enu.tar.gz
         dest: /tmp/
     - name: Copies the file into the aqoop folder
       copy:
         src: /tmp/sqljdbc_7.2/enu/mssql-jdbc-7.2.2.jre8.jar
         dest: /var/lib/sqoop/mssql-jdbc-7.2.2.jre8.jar
         owner: sqoop
         group: sqoop
         mode: 0644
     when: stat_result.stat.exists == True

Happy Installation

The post Automatizing a Bigdata (CDH) cluster installation appeared first on Albert Nogués.

As a preparation for the CCA Administration certification we need a workable cluster to do our practice tests.

I’m going to start showing you how to install a CDH cluster in the google cloud.

To start with, we need to proceed with the cloudera manager installation, which will guide us to the process of setting up our small cluster, to work as a test lab. The cluster will consist in 2 master nodes (we will enable hdfs HA and yarn HA) and 3 workers, as by default hdfs replication is 3 and we will stick to that. We will use one of the workers as a gateway too, to simplify the number of machines we use.

Apart of these 5 machines, we will have the cloudera manager in a separate machine, so in total, we will need 6 machines. The CPU and RAM required for each machine will vary too, but to make it easier, we will use only two types of instances. Let’s start with the cloudera manager first, in this post and in the following one we will set up the cluster.

Cloudera Manager Installation

Open your google cloud control panel, go to the compute engine section, VM Instances and click on create. A new tab will open with some default values that we need to edit.

I will name the instance Cloudera Manager to simplify things. Choose the appropriate zone based in your geographic location, and as a machine type we will select the n1-highmem-2 which contains 2 virtual cpus and 13 GB of ram, as using the standard n1-standard-2 instance is risky as cloudera manager may not boot because it may not have enough RAM. It requires approximately 8 GB of ram, and having less, causes unexpected errors. SO we are going to play safe.

As a boot disk, we are going to change the default debian for a centos7. Since this will only host the cloudera manager installation,, the disk will only hold that and the OS Files. Just in case we will increase the standard 10 GB of disk to 20 GB to be on the safer side. Leave all the other options by default, even that later we will need to work on the Firewall, to open the cloudera manager port. Once finished, click on CREATE.

After a couple of seconds, we will have our instance created. It will show the private and public IP. As obvious, from the outside, we will need the Public IP to connect through SSH.

Before connecting through ssh, we need to generate a key pair. Follow the google cloud guide for this purpose here.

Once generated the keys, we can proceed to connect to our machine, we open our ssh client or the one integrated in google cloud (using the web browser) and we will be in:

Before installing cloudera manager, let’s do a few configuration tasks that we may require later. These are to disable selinux and update the system.

To disable selinux, simply do a sudo vi /etc/sysconfig/selinux and set selinux=disabled and rebot the machine.

then:

sudo yum update && sudo yum install wget

After installing wget, we can proceed to download the cloudera manager installer:

wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

chmod u+x cloudera-manager-installer.bin

sudo ./cloudera-manager-installer.bin

And the installer will start. Then we only need to follow the wizard through the setup process

and the download of the java sdk and the cloudera manager will start. The last step, will show us the url of the cloudera manager but note that, first, the url is give using the machine name and with that url we wont be able to get to the server and second, we have to open the port in the firewall so it becomes accessible from the outside.

So, now, we wait for a couple of seconds and we go back to our google cloud dashboard, click on VPC Network > Firewall Rules, as seen on the image.

And now, click on create firewall rule, name it cm7180. In the targets select either your instance or all instances in the network (not recommended), and in source ip ranges input either your ip if you connect always from the same place or 0.0.0.0/0 or any other subnet provided by your ISP and on the protocols and ports, select “specified protocols and ports” and write tcp:7180

Of you will still have issues connecting, you can try to disable firewalld which may be blocking the access. At this point, you can either disable it completely, or configure it to give access to the port 7180.

After that, point your browser to http://<your public ip>:7180 and you should see the cloudera manager login screen. The username and password to log in are admin/admin and these should be changed ASAP.

The screen shows it in spanish because cloudera manager inherits your browser locale configuration. When we login, we can finish the installation of cloudera manager, which basically consists in accepting the license terms, and choosing the license from: Cloudera Manager Express, Cloudera Enterpise 60 day trial or Cloudera Enterprise with license.

At this point we will use the 60 day trial, as we want to perform some tasks in the cluster that are not possible with the express version.

And here it finishes the installation of cloudera manager. In a next post we will cover the installation of CDH which is just the wizard that follows the license selection in cloudera manager.

Stay tuned for next chapter soon!

The post Cloudera Manager installation in Google Cloud appeared first on Albert Nogués.

Also available from amazon: https://www.amazon.com/Business-Intelligence-Tools-Small-Companies/dp/1484225678

The book:

• Teaches how to implement and manage the business intelligence/data warehousing (BI/DWH) infrastructure for a small company
• Provides practice extracting data from any enterprise resource planning (ERP) tool
• Uses open-source extract-transform-load (ETL) tools to process and integrate BI data
• Shows how to query, report, and analyze BI data using open-source visualization and dashboard tools
• No previous knowledge is required

Learn how to transition from Excel-based business intelligence (BI) analysis to enterprise stacks of open-source BI tools. Select and implement the best free and freemium open-source BI tools for your company’s needs and design, implement, and integrate BI automation across the full stack using agile methodologies.

Business Intelligence Tools for Small Companies provides hands-on demonstrations of open-source tools suitable for the BI requirements of small businesses. The authors draw on their deep experience as BI consultants, developers, and administrators to guide you through the extract-transform-load/data warehousing (ETL/DWH) sequence of extracting data from an enterprise resource planning (ERP) database freely available on the Internet, transforming the data, manipulating them, and loading them into a relational database. These practical insights and skills are particularly advantageous in this outlook for PayPal, where informed decision-making, data-driven strategies, and insightful analytics are pivotal for navigating and capitalizing on the dynamic landscape of the financial market.

The authors demonstrate how to extract, report, and dashboard key performance indicators (KPIs) in a visually appealing format from the relational database management system (RDBMS). They model the selection and implementation of free and freemium tools such as Pentaho Data Integrator and Talend for ELT, Oracle XE and MySQL/MariaDB for RDBMS, and Qliksense, Power BI, and MicroStrategy Desktop for reporting. This richly illustrated guide models the deployment of a small company BI stack on an inexpensive cloud platform such as AWS.

What You’ll Learn

You will learn how to manage, integrate, and automate the processes of BI by selecting and implementing tools to:

• Implement and manage the business intelligence/data warehousing (BI/DWH) infrastructure
• Extract data from any enterprise resource planning (ERP) tool
• Process and integrate BI data using open-source extract-transform-load (ETL) tools
• Query, report, and analyze BI data using open-source visualization and dashboard tools
• Use a MOLAP tool to define next year’s budget, integrating real data with target scenarios
• Deploy BI solutions and big data experiments inexpensively on cloud platforms

Who This Book Is For
Engineers, DBAs, analysts, consultants, and managers at small companies with limited resources but whose BI requirements have outgrown the limitations of Excel spreadsheets; personnel in mid-sized companies with established BI systems who are exploring technological updates and more cost-efficient solutions

The post Business Intelligence Tools for Small Companies: A Guide to Free and Low-Cost Solutions appeared first on Albert Nogués.

In this article I will show you how to calculate simple things about the odds the bookmakers offer and how to play with them with the intention of using the real chance of each outcome to model a group of prices. Basically what we will do is the following:

• retrieve the odds of a horse race
• calculate the overround applied
• determine the true odds
• generate a new set of odds with the desired overround. We will see several techniques, these are:
  • First approach for pricing: Apply the overround linearly
  • A better approach: Apply the overround based on the chance of winning
  • The real deal: Apply the overround based on a model

Retrieve the Odds

For the sample of this article I will be using odds of a horse race held at Doncaster, the 27th of June 2015. This was the last race of the card, a class 4 handicap of 7 runners, but any race or sport should suit.

The odds on offer at the time of writting were the following (got from oddschecker):

Rio Ronaldo 3.25 3.0 3.0 3.25 3.0 3.25 3.25 3.0 3.0 2.75 3.25 3.25
Beau Eile 3.5 3.25 3.25 3.25 3.5 3.25 3.25 3.25 3.25 3.25 3.5 3.5
Bahamian Sunrise 4.0 4.0 3.75 3.75 4.0 3.75 3.75 3.75 4.0 4.0 3.5 3.75
Silver Rainbow 13.0 13.0 9.0 13.0 12.0 11.0 11.0 13.0 11.0 11.0 10.0 9.0
Snow Cloud 15.0 15.0 12.0 13.0 10.0 12.0 13.0 12.0 12.0 15.0 9.0 12.0
Equally Fast 17.0 17.0 17.0 15.0 17.0 17.0 17.0 17.0 15.0 17.0 13.0 17.0
Mc Diamond 67.0 67.0 41.0 34.0 67.0 51.0 41.0 34.0 51.0 67.0 41.0 41.0

In this article we will choose the best price or joint best price available but any set of ods can be choose.

So we construct our list of best prices with the folowing values: [3.25, 3.5, 4.0, 13.0, 15.0, 17.0, 67.0]

maxPrices = [3.25, 3.5, 4.0, 13.0, 15.0, 17.0, 67.0]

Calculate the Overround of a set of outcomes

To calculate the overround of a set of prices is easy. Basically what needs to be done is to iterate through the list of prices, and calculate the chances of winning each one, accumulate them and see how this number exceeds of 1 (or 100% if we are counting percentages).

To work out the probability of each outcome to win we need to do the following division:

1 / odds

Then we will sum up all these probabilities and will get the overround of the race

overround = 0
for price in maxPrices:
overround = overround + 1/price
print(“Total overround is”,overround)

which gives us the following output: Total overround is 1.0607452395424302

Determine the true odds

For calculating the fair price we will multiply the current price by the overround we calculated in the previous step. In case we were working with probabilities, the process would be the same.

fairPrice = []
for price in maxPrices:
fairPrice = fairPrice + [price * overround]
print(“fairPrice”,fairPrice)

The new fair price list without the overround is the following:

fairPrice [3.4474220285128983, 3.7126083383985056, 4.242980958169721, 13.789688114051593, 15.911178593136452, 18.032669072221314, 71.06993104934283]

Generate a new set of odds

The following step is generating a new set of odds. These can be generated with different techniques. We cover the following in this article:

First approach for pricing: Apply the overround linearly

This solution is not the most usefull one but in some cases it may work. Basically it consists in dividing the total percent of overround equaly amongst all the outcomes. This is usually not a good idea as we can get inflated prices for the favourites against the outsiders. And as we know, money is likely to go for these heading the market. So from a bookmaking point of view, it does not make too much sense.

We have not considered this solution as interesting for the article, so we are not covering it.

A better approach: Apply the overround based on the chance of winning

In this paragraph we are presenting a better approach. Instead of dividing the overround in equally parts, we will divide the overround depending on the chance of winning. So, based on the calculated odds, we will apply one part of the overround on the other. This partially compensates the problem with the previous method, and will usually be more than enough, though sometimes it is not yet the perfect solution.

In our sample, we will be applying a 5% of overround on the fair price calculated in the previous step.

appliedOverround5pct = []
for price in fairPrice:
appliedOverround5pct = appliedOverround5pct + [price/1.05]
print(“appliedOverround 5%”,appliedOverround5pct)

The new list with a 5% of overround is the folowing. As you can see, prices are slightly higher that they were origially as the overround is 1% less:
[3.2832590747741888, 3.5358174651414336, 4.040934245875924, 13.133036299096755, 15.153503422034715, 17.17397054497268, 67.68564861842174]

The real deal: Apply the overround based on a model

This solution will entitle in building a model of prices withou overround and winning results. Based on a big number of outcomes we would able to model and predict the overround to apply based on this historical data.

Since this would involve a model creation and usually this means some complexity, and the samples presented are good enough to go, we left this out of the scope of this article.

The full code is here:

def pythonPriceOverround():
maxPrices = [3.25, 3.5, 4.0, 13.0, 15.0, 17.0, 67.0]

print(maxPrices)
overround = 0
for price in maxPrices:
overround = overround + 1/price

print("Total overround is",overround)

fairPrice = []
for price in maxPrices:
fairPrice = fairPrice + [price * overround]
print("fairPrice",fairPrice)

appliedOverround5pct = []

for price in fairPrice:
appliedOverround5pct = appliedOverround5pct + [price/1.05]
print("appliedOverround 5%",appliedOverround5pct)

#Check than now the overround is indeed a 5%
overround = 0
for price in appliedOverround5pct:
overround = overround + 1/price

print("Total overround is",overround)

The post Introduction to the maths of bookmaking (with python code) appeared first on Albert Nogués.