.gitignore

+# python notebook checkpoints
+materials/.ipynb_checkpoints/
\ No newline at end of file

README.md

@@ -6,8 +6,8 @@ Denmark Technical University.
 ## Workshop objective
 
 To help Matlab-experienced users begin coding in Python. This includes a review
-of syntax differences, along with a brief introduction of the most commonly
-used libraries, and a quick overview of PEP8 coding conventions.
+of how to import modules along with a brief introduction of the most commonly
+used packages (NumPy, Matplotlib, and pandas).
 
 ## Who should come
 
@@ -23,13 +23,11 @@ arranging a new workshop.
 
 ## Topic outline
 
-- "Big-picture" differences
-- Syntax differences
+- Import modules/packages  
 - Useful packages:
     - NumPy/SciPy
     - Pandas
     - Matplotlib
-- PEP8 coding convention
 
 ## Prerequisites
 

materials/1c-exercise_importing_module_solution.py

+"""Solutions to first exercise on import modules
+
+Author
+------
+Jenni Rinker
+rink@dtu.dk
+"""
+
+import hello_world
+import hello_world as hw
+from hello_world import greetings
+from hello_world import greetings as gr
\ No newline at end of file

materials/2a-tutorial_numpy_array_basics.ipynb

+%% Cell type:markdown id: tags:
+
+# Basics of NumPy arrays
+
+%% Cell type:markdown id: tags:
+
+Let's quickly go over NumPy arrays, which are "objects" defined in the NumPy package that function extremely similar to Matlab's matrices. Any defined array comes with a series of "attributes" (i.e., values nested inside the object) and "methods" (i.e., functions nested inside the object) that are built into that object. All of the attributes and methods associated with a NumPy array for version 1.13 can be found [here](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.html).
+
+So, let's mess around with this for a bit.
+
+%% Cell type:markdown id: tags:
+
+## Preliminaries
+
+%% Cell type:markdown id: tags:
+
+In order to use any NumPy functions, we first need to import the package (assuming you already have it installed using conda/pip/some package manager). It's traditional to import NumPy as "np" so we don't have to type as much later. Also, note that package imports are case-sensitive.
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+```
+
+%% Cell type:markdown id: tags:
+
+## Creating and slicing NumPy arrays
+
+%% Cell type:markdown id: tags:
+
+Unlike Matlab, arrays must be initialized before their values can be assigned. You can specify the datatype when you initialize (e.g., float, integer, string, complex, etc. -- default is float).
+
+%% Cell type:code id: tags:
+
+``` python
+a = np.empty((5, 2), dtype=float)  # without this line, the following lines would break!
+for i in range(5):
+    a[i, 0] = i ** 2  # assign square(i) to col 0
+    a[i, 1] = 2 * i  # assign double(i) to col 1
+a  # jupyter will print the last line if it's not assigned to anything
+```
+
+%% Cell type:markdown id: tags:
+
+Array slicing is extremely similar to Matlab, except for a few things:
+1. Slices are done with square brackets (curved brackets are only for function inputs)
+2. Indexing starts from 0
+3. The end index you specifiy is not included in the slice
+4. There is no `end`, but you can just omit the last index or use negative indexing
+5. Omitting the beginning index implies starting at 0; omitting the end implies ending at the end.
+
+%% Cell type:code id: tags:
+
+``` python
+print(a[0:3, 0])  # print elements 0, 1, and 2 in first column
+print(a[:3, 0])  # same as above, but we've omitted the zero in the indexing this time
+print(a[-2:, 1])  # print last two elements in second column
+print(a[:, 0])  # print all elements in first column
+print(a[[0, 1, 3], 1])  # print 1st, 2nd, and 4th elements in first column (uses broadcasting, more on that later)
+print(a[[0, 1, 3], [0, 1, 0]])  # print elements [0,0], [1,1], and [3,0]
+```
+
+%% Cell type:markdown id: tags:
+
+You can create arrays with as many dimensions as you like
+
+%% Cell type:code id: tags:
+
+``` python
+b = np.random.rand(10, 3, 20, 4)  # dim1 = 10, dim2 = 3, etc.
+print(b.shape)  # 'shape' is an attribute, it returns a tuple
+```
+
+%% Cell type:markdown id: tags:
+
+And you can use a lot of the methods "built into" the array. Remember, these are documented [here](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.html).
+
+%% Cell type:code id: tags:
+
+``` python
+print(a.mean())  # mean of entire matrix
+print(a.max(axis=0))  # max of each column
+print(a.min(axis=1))  # min of each row
+print(a.argmax())  # this is applied to the raveled matrix, so it returns an integer
+print(a.ravel()[a.argmax()])  # ...so we can either apply the index to the flattened matrix
+print(np.unravel_index(a.argmax(), a.shape))  # ...or convert the integer index to a tuple
+```
+
+%% Cell type:markdown id: tags:
+
+Wait...what do we mean by a flattened array? And what does `ravel` do?
+
+In short, `flatten` will return a copy of the array, but `ravel` (and `reshape`) will not. This is important, because if you write something to the output of `flatten`, you will not update the original array.
+
+%% Cell type:code id: tags:
+
+``` python
+print(a.ravel(), '...line 1')  # this is just a (contiguous) view of the array, it does not return a copy
+print(a.flatten(), '...line 2')  # this returns a copy of the array
+print(a.reshape((-1)), '...line 3')  # this will return a (non-contiguous) view of the array
+a.flatten()[8] = np.nan  # assign a nan to the output of flatten
+print(a.ravel(), '...line 4')  # ...but a is unchanged!
+a.ravel()[8] = np.nan  # assign a nan to the output of ravel
+print(a.ravel(), '...line 5')  # ...and now a is changed!
+a.ravel()[8] = 16.0  # assign the correct value back to a
+```
+
+%% Cell type:markdown id: tags:
+
+## Elementwise multiplication and array broadcasting
+
+%% Cell type:markdown id: tags:
+
+Matrix multiplication is one of the most common things we do in our analyses, but if you're not careful you can be tripped up by NumPy's matrix operations.
+
+%% Cell type:code id: tags:
+
+``` python
+my_mat = np.array([[0, 1], [2, 3]])
+v1 = np.arange(2)  # 1d array of [0, 1]
+v2 = np.arange(2).reshape((-1, 1))  # v1 but as a column vector (note the '-1' indicates to infer the size along that axis)
+v3 = np.arange(2).reshape((1, -1))  # v1 but as a row vector
+```
+
+%% Cell type:markdown id: tags:
+
+Print their shapes just for clarity...
+
+%% Cell type:code id: tags:
+
+``` python
+print(my_mat.shape)
+print(v1.shape)  # note that the shape tuple for v1 has only one number, whereas v2 and v3 have two numbers
+print(v2.shape)
+print(v3.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+What happens if we multiply things togeter? $\mathrm{my\_mat} \cdot v_3$ should break, right?
+
+%% Cell type:code id: tags:
+
+``` python
+print(my_mat * v1)
+print(my_mat * v2)
+print(my_mat * v3)
+```
+
+%% Cell type:markdown id: tags:
+
+Wait...this doesn't make sense at all! How are all these output matrices $2\times2$?
+
+**Answer**: NumPy generally tries to do everything element-wise. So, `*` is element-wise multiplication.
+
+NumPy handles the mismatch in dimensions by "broadcasting" arrays into the other dimensions to try and make dimensions match so it can do things element-wise.
+
+**Broadcasting is done in two ways:**
+1. If a shape tuple has too few elements for an operation (e.g., `(2,)` vs. `(2,2)`), then NumPy will prepend "1" to the shape tuple until it has the right number of elements.
+2. If there are the right number of elements in the shape tuple, but 1) the shapes don't match and 2) one of the elements in the shape tuple is a 1, NumPy tries to repeat the array until it matches the size of the other array.
+
+If NumPy cannot get the dimensions to match using the above two steps, then it will throw an error.
+
+%% Cell type:markdown id: tags:
+
+Let's now explain what's going on in our example a little more clearly.
+
+#### Broadcasting of v1 array
+- v1 starts out as the following arrray: `[0, 1]` (shape `(2,)`)
+- A 1 is preprended to the shape tuple so it has the right number of elements: `[[0, 1]]` (shape `(1, 2)`)
+- The array is then tiled along the 0 axis (down rows) until its shape matches `my_mat`: `[[0, 1], [0, 1]]` (shape `(2, 2)`)
+- This broadcasted array is then multiplied by `my_mat` element-wise
+
+#### Broadcasting of v2 array
+- v2 starts out as the following arrray: `[[0], [1]]` (shape `(2, 1)`)
+- The array is then tiled along the 1 axis (along columns) until its shape matches `my_mat`: `[[0, 0], [1, 1]]` (shape `(2, 2)`)
+- This broadcasted array is then multiplied by `my_mat` element-wise
+
+#### Broadcasting of v3 array
+- v3 starts out as the following arrray: `[[0, 1]]` (shape `(1, 2)`)
+- The array is then tiled along the 0 axis (down rows) until its shape matches `my_mat`: `[[0, 1], [0, 1]]` (shape `(2, 2)`)
+- This broadcasted array is then multiplied by `my_mat` element-wise
+
+%% Cell type:markdown id: tags:
+
+## Matrix multiplication
+
+%% Cell type:markdown id: tags:
+
+*I'm not sure how that's useful to know*, you are thinking. *I normally do matrix multiplication*. Well, let's figure that out, eh?
+
+%% Cell type:code id: tags:
+
+``` python
+print(my_mat @ v1)  # note this returns a 1D array
+print(my_mat @ v2)  # ...while this returns a 2D array
+print(np.dot(my_mat, v2))  # you can also use the numpy function
+# print(my_mat @ v3)  # this breaks with a 'dimension mismatch' error, just as we would expect
+```
+
+%% Cell type:markdown id: tags:
+
+Well. That was easy.

materials/2b-exercise_numpy.ipynb

+%% Cell type:markdown id: tags:
+
+# NumPy Exercise: Correlated Gaussian Random Variables
+
+%% Cell type:markdown id: tags:
+
+In many situations, we need to have a series of correlated Gaussian random variables, which can then be transformed into other distributions of interest (uniform, lognormal, etc.). Let's see how to do that with NumPy in Python.
+
+### Given:
+
+|Variable | Value | Description |
+| ---: | :---: | :--- |
+|`n_real` | `1E6` | number of realizations|
+|`n_vars` | 3 | number of variables to correlate|
+|`cov` | `[[ 1. ,  0.2,  0.4], [ 0.2,  0.8,  0.3], [ 0.4,  0.3,  1.1]]` | covariance matrix|
+
+### Theory
+
+The procedure for generating correlated Gaussian is as follows:
+1. Sample `[n_vars x n_real]` (uncorrelated) normal random variables
+2. Calculate `chol_mat`, the Cholesky decomposition of the covariance matrix
+3. Matrix-multiply your random variables with `chol_mat` to produce a `[n_vars x n_real]` array of correlated Gaussian variables
+
+### Exercise
+
+Do the following:
+1. Fill in the blank cells below so that the code follows the theory outlined above.
+2. Calculate the variances of the three samples of random variables. Does it match the diagonal of the covariance matrix?
+3. Calculate the correlation coefficient between the first and second random samples. Does it match `cov[0, 1]`?
+
+### Hints
+
+- In the arrays of random variables, each row `i` corresponds to a *sample* of random variable `i` (just FYI).
+- Google is your friend :)
+
+%% Cell type:code id: tags:
+
+``` python
+  # import any needed modules here
+```
+
+%% Cell type:code id: tags:
+
+``` python
+n_real =  # number of realizations
+n_vars =  # number of random variables we want to correlate
+cov =  # covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+unc_vars =  # create [n_vars x n_real] array of uncorrelated (unc) normal random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chol_mat =  # calculate the cholesky decomposition of the covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+cor_vars =  # [n_vars x n_real] array of correlated (cor) random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+  # calculate variances of each sample of random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+  # calculate the correlation coefficient between the first and second random samples
+```

materials/2c-exercise_numpy_solutions.ipynb

+%% Cell type:markdown id: tags:
+
+# NumPy Exercise: Correlated Gaussian Random Variables
+
+%% Cell type:markdown id: tags:
+
+In many situations, we need to have a series of correlated Gaussian random variables, which can then be transformed into other distributions of interest (uniform, lognormal, etc.). Let's see how to do that with NumPy in Python.
+
+### Given:
+
+|Variable | Value | Description |
+| ---: | :---: | :--- |
+|`n_real` | `1E6` | number of realizations|
+|`n_vars` | 3 | number of variables to correlate|
+|`cov` | `[[ 1. ,  0.2,  0.4], [ 0.2,  0.8,  0.3], [ 0.4,  0.3,  1.1]]` | covariance matrix|
+
+### Theory
+
+The procedure for generating correlated Gaussian is as follows:
+1. Sample `[n_vars x n_real]` (uncorrelated) normal random variables
+2. Calculate `chol_mat`, the Cholesky decomposition of the covariance matrix
+3. Matrix-multiply your random variables with `chol_mat` to produce a `[n_vars x n_real]` array of correlated Gaussian variables
+
+### Exercise
+
+Do the following:
+1. Fill in the blank cells below so that the code follows the theory outlined above.
+2. Calculate the variances of the three samples of random variables. Does it match the diagonal of the covariance matrix?
+3. Calculate the correlation coefficient between the first and second random samples. Does it match `cov[0, 1]`?
+
+### Hints
+
+- In the arrays of random variables, each row `i` corresponds to a *sample* of random variable `i` (just FYI).
+- Google is your friend :)
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np  # import any needed modules here
+```
+
+%% Cell type:code id: tags:
+
+``` python
+n_real = int(1E6)  # number of realizations
+n_vars = 3  # number of random variables we want to correlate
+cov = np.array([[ 1. ,  0.2,  0.4], [ 0.2,  0.8,  0.3], [ 0.4,  0.3,  1.1]])  # covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+unc_vars = np.random.randn(n_vars, n_real)  # create [n_vars x n_real] array of uncorrelated (unc) normal random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chol_mat = np.linalg.cholesky(cov)  # calculate the cholesky decomposition of the covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+cor_vars = chol_mat @ unc_vars  # [n_vars x n_real] array of correlated (cor) random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+cor_vars.var(axis=1)  # calculate variances of each sample of random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+np.corrcoef(cor_vars[0, :], cor_vars[1, :])  # calculate the correlation coefficient between the first and second random samples
+```

materials/3b-exercise_matplotlib.ipynb

+%% Cell type:markdown id: tags:
+
+# Matplotlib Exercise: Visualizing Correlated Gaussian Random Variables
+
+%% Cell type:markdown id: tags:
+
+Now that we know how to generate correlated random variables, let's visualize them.
+
+### Exercise
+
+Make the following plots. All plots must have x and y labels, titles, and legends if there is more than one dataset in the same axes.
+
+1. Overlaid histograms of your samples of uncorrelated random variables with 30 bins (use `histtype='step'`)
+2. A scatterplot of $X_2$ vs $X_1$ with marker size equal to 2. Overlay the the theoretical line ($y=x$) in a black, dashed line.
+3. Overlaid histograms of your samples of correlated random variables with 30 bins (use `histtype='step'`)
+
+### Hints
+
+- In the arrays of random variables, each row `i` corresponds to a *sample* of random variable `i` (just FYI).
+- Google is your friend :)
+
+%% Cell type:code id: tags:
+
+``` python
+import matplotlib.pyplot as plt  # need to import matplotlib, of course
+import numpy as np  # import any needed modules here
+```
+
+%% Cell type:code id: tags:
+
+``` python
+n_real = int(1E6)  # number of realizations
+n_vars = 3  # number of random variables we want to correlate
+cov = np.array([[ 1. ,  0.2,  0.4], [ 0.2,  0.8,  0.3], [ 0.4,  0.3,  1.1]])  # covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+unc_vars = np.random.randn(n_vars, n_real)  # create [n_vars x n_real] array of uncorrelated (unc) normal random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chol_mat = np.linalg.cholesky(cov)  # calculate the cholesky decomposition of the covariance matrix
+```
+
+%% Cell type:code id: tags:
+
+``` python
+cor_vars = chol_mat @ unc_vars  # [n_vars x n_real] array of correlated (cor) random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+cor_vars.var(axis=1)  # calculate variances of each sample of random variables
+```
+
+%% Cell type:code id: tags:
+
+``` python
+np.corrcoef(cor_vars[0, :], cor_vars[1, :])  # calculate the correlation coefficient between the first and second random samples
+```
+
+%% Cell type:markdown id: tags:
+
+## Plot 1: Histogram of Uncorrelated Variables
+
+%% Cell type:markdown id: tags:
+
+Make a plot with overlaid histograms of your samples of uncorrelated random variables with 30 bins (use histtype='step').
+
+%% Cell type:code id: tags:
+
+``` python
+  # insert code here
+```
+
+%% Cell type:markdown id: tags:
+
+## Plot 2: Scatterplot of X2 vs. X1
+
+%% Cell type:markdown id: tags:
+
+Make a scatterplot of $X_2$ vs $X_1$ with marker size equal to 2. Overlay the the theoretical line ($y=x$) in a black, dashed line.
+
+%% Cell type:code id: tags:
+
+``` python
+  # insert code here
+```
+
+%% Cell type:markdown id: tags:
+
+## Plot 3: Histogram of Correlated Variables
+
+%% Cell type:markdown id: tags:
+
+Make a plot with overlaid histograms of your samples of uncorrelated random variables with 30 bins (use histtype='step').
+
+%% Cell type:code id: tags:
+
+``` python
+  # insert code here
+```

materials/4b-exercise_pandas.ipynb

+%% Cell type:markdown id: tags:
+
+# Testing You in Pandas
+
+%% Cell type:markdown id: tags:
+
+Given the Risø V52 met mast data, let's select and make some plots.
+
+### Exercises
+
+Make the following plots:
+1. For a wind speed sector from 250 to 340 degrees, plot a histogram of the mean cup anemometer mean wind speed at 70 m.
+2. Scatter plot the turbulence intensity versus the mean wind speed for two heights (your choice). Overlaid plots are preferred, but if you can't figure out how to do it, then separate plots are fine.
+
+Be sure to add axis labels, legends, titles, etc., where applicable.
+
+%% Cell type:markdown id: tags:
+
+## Preliminaries
+
+%% Cell type:markdown id: tags:
+
+As always, we must first import the modules we want to use before we can write any code. I'm also setting the jupyter matploblib option to be interactive, as we can do in notebooks.
+
+%% Cell type:code id: tags:
+
+``` python
+% matplotlib notebook
+import matplotlib.pyplot as plt
+import pandas as pd
+```
+
+%% Cell type:markdown id: tags:
+
+Load the mean and std dev `.csv` files to dataframes and concatenate them.
+
+%% Cell type:code id: tags:
+
+``` python
+dfs = []
+for csv_name, suffix in zip(['demo_risoe_data_means.csv', 'demo_risoe_data_stdvs.csv'], ['_mean', '_stdv']):
+    df_path = f'data/{csv_name}'
+    df = pd.read_csv(df_path)  # load csv to dataframe
+    df['name'] = pd.to_datetime(df['name'].astype(str), format='%Y%m%d%H%M')  # convert name to datetime
+    df.set_index('name', inplace=True)  # set name as index
+    df = df.add_suffix(suffix)
+    dfs.append(df)
+met_df = pd.concat(dfs, axis=1);  # suppress jupyter output with semicolon
+```
+
+%% Cell type:markdown id: tags:
+
+## Exercise 1: Wind Speed Histogram for Sector
+
+For a wind speed sector from 250 to 340 degrees, plot a histogram of the mean cup anemometer mean wind speed at 70 m.
+
+%% Cell type:code id: tags:
+
+``` python
+  # insert code here!
+```
+
+%% Cell type:markdown id: tags:
+
+## Exercise 2: TI vs U
+
+Scatter plot the turbulence intensity versus the mean wind speed for two heights (your choice).
+
+%% Cell type:code id: tags:
+
+``` python
+  # insert code here!
+```

materials/4c-exercise_pandas_solutions.ipynb

+%% Cell type:markdown id: tags:
+
+# Testing You in Pandas
+
+%% Cell type:markdown id: tags:
+
+Given the Risø V52 met mast data, let's select and make some plots.
+
+### Exercises
+
+Make the following plots:
+1. For a wind speed sector from 250 to 340 degrees, plot a histogram of the mean cup anemometer mean wind speed at 70 m.
+2. Scatter plot the turbulence intensity versus the mean wind speed for two heights (your choice). Overlaid plots are preferred, but if you can't figure out how to do it, then separate plots are fine.
+
+Be sure to add axis labels, legends, titles, etc., where applicable.
+
+%% Cell type:markdown id: tags:
+
+## Preliminaries
+
+%% Cell type:markdown id: tags:
+
+As always, we must first import the modules we want to use before we can write any code. I'm also setting the jupyter matploblib option to be interactive, as we can do in notebooks.
+
+%% Cell type:code id: tags:
+
+``` python
+% matplotlib notebook
+import matplotlib.pyplot as plt
+import pandas as pd
+```
+
+%% Cell type:markdown id: tags:
+
+Load the mean and std dev `.csv` files to dataframes and concatenate them.
+
+%% Cell type:code id: tags:
+
+``` python
+dfs = []
+for csv_name, suffix in zip(['demo_risoe_data_means.csv', 'demo_risoe_data_stdvs.csv'], ['_mean', '_stdv']):
+    df_path = f'data/{csv_name}'
+    df = pd.read_csv(df_path)  # load csv to dataframe
+    df['name'] = pd.to_datetime(df['name'].astype(str), format='%Y%m%d%H%M')  # convert name to datetime
+    df.set_index('name', inplace=True)  # set name as index
+    df = df.add_suffix(suffix)
+    dfs.append(df)
+met_df = pd.concat(dfs, axis=1);  # suppress jupyter output with semicolon
+```
+
+%% Cell type:markdown id: tags:
+
+## Exercise 1: Wind Speed Histogram for Sector
+
+For a wind speed sector from 250 to 340 degrees, plot a histogram of the mean cup anemometer mean wind speed at 70 m.
+
+%% Cell type:code id: tags:
+
+``` python
+ax = met_df[(met_df.Wdir_41m_mean >= 250) & (met_df.Wdir_41m_mean <= 340)].plot(y='Wsp_70m_mean', kind='hist')
+ax.set_xlabel('Cup Anem. Wind Speed @ 70 m')
+ax.legend().set_visible(False)  # hide the legend since we have an x axis
+```
+
+%% Output
+
+
+
+%% Cell type:markdown id: tags:
+
+## Exercise 2: TI vs U
+
+Scatter plot the turbulence intensity versus the mean wind speed for two heights (your choice).
+
+%% Cell type:code id: tags:
+
+``` python
+heights = [44, 57, 70]  # I'll do three because it's fun!
+colors = ['r', 'g', 'b']
+ax = None
+for i_ht, ht in enumerate(heights):
+    met_df[f'TI_{ht}m'] = met_df[f'Wsp_{ht}m_stdv'] / met_df[f'Wsp_{ht}m_mean']
+    ax = met_df.plot(x=f'Wsp_{ht}m_mean', y=f'TI_{ht}m', kind='scatter',
+                     c=colors[i_ht], s=2, label=f'{ht} m', ax=ax)
+ax.legend()
+ax.set_xlabel('Mean Wind Speed')
+ax.set_ylabel('Turbulence Intensity');
+```
+
+%% Output
+
+
+
+%% Cell type:code id: tags:
+
+``` python
+```