\subsection{Recall}
\label{chp:fundamentals:sec:metrics:subsec:Recall}

The next metric we introduce is called \textit{recall} and is defined \textit{"as the probability of detecting an item given that it is relevant"} \parencite{Zhu:2004}.
Its formula is given by \cref{eq:recall}.

\begin{equation}\label{eq:recall}
    rec = \frac{\acp{TP}}{\acp{TP}+\acp{FN}}
\end{equation}

This metric expresses the performance of an algorithm in terms of how many of all relevant items it manages to select.
However, irrelevant items which may have been incorrectly selected are not be penalized.
Therefore, an algorithm which simply selects \textit{all} available items always achieves a recall $rec=1$.
For the data shown in \cref{fig:metrics:tp_vis}, the corresponding algorithm achieves a recall of $rec = \frac{3}{3+9} = 0.25$.


\chapter{实验代码}

\section{Shamir 秘密共享}
\label{appendix:Shamir}

shamir.py

\begin{lstlisting}[language = Python]
from Crypto.Util.py3compat import is_native_int
from Crypto.Util import number
from Crypto.Util.number import long_to_bytes, bytes_to_long
from Crypto.Random import get_random_bytes as rng

def _mult_gf2(f1, f2):
    """Multiply two polynomials in GF(2)"""

    # Ensure f2 is the smallest
    if f2 > f1:
        f1, f2 = f2, f1
    z = 0
    while f2:
        if f2 & 1:
            z ^= f1
        f1 <<= 1
        f2 >>= 1
    return z

def _div_gf2(a, b):
    """
    Compute division of polynomials over GF(2).
    Given a and b, it finds two polynomials q and r such that:

    a = b*q + r with deg(r)<deg(b)
    """

    if (a < b):
        return 0, a

    deg = number.size
    q = 0
    r = a
    d = deg(b)
    while deg(r) >= d:
        s = 1 << (deg(r) - d)
        q ^= s
        r ^= _mult_gf2(b, s)
    return (q, r)

class _Element(object):
    """Element of GF(2^128) field"""

    # The irreducible polynomial defining this field is 1+x+x^2+x^7+x^128
    irr_poly = 1 + 2 + 4 + 128 + 2 ** 128

    def __init__(self, encoded_value):
        """Initialize the element to a certain value.

        The value passed as parameter is internally encoded as
        a 128-bit integer, where each bit represents a polynomial
        coefficient. The LSB is the constant coefficient.
        """

        if is_native_int(encoded_value):
            self._value = encoded_value
        # elif len(encoded_value) == 16:
        else:
            self._value = bytes_to_long(encoded_value)
        # else:
        #     raise ValueError("The encoded value must be an integer or a 16 byte string")

    def __eq__(self, other):
        return self._value == other._value

    def __int__(self):
        """Return the field element, encoded as a 128-bit integer."""
        return self._value

    def encode(self):
        """Return the field element, encoded as a 16 byte string."""
        return long_to_bytes(self._value)

    def __mul__(self, factor):

        f1 = self._value
        f2 = factor._value

        # Make sure that f2 is the smallest, to speed up the loop
        if f2 > f1:
            f1, f2 = f2, f1

        if self.irr_poly in (f1, f2):
            return _Element(0)

        mask1 = 2 ** 128
        v, z = f1, 0
        while f2:
            # if f2 ^ 1: z ^= v
            mask2 = int(bin(f2 & 1)[2:] * 128, base=2)
            z = (mask2 & (z ^ v)) | ((mask1 - mask2 - 1) & z)
            v <<= 1
            # if v & mask1: v ^= self.irr_poly
            mask3 = int(bin((v >> 128) & 1)[2:] * 128, base=2)
            v = (mask3 & (v ^ self.irr_poly)) | ((mask1 - mask3 - 1) & v)
            f2 >>= 1
        return _Element(z)

    def __add__(self, term):
        return _Element(self._value ^ term._value)

    def inverse(self):
        """Return the inverse of this element in GF(2^128)."""

        # We use the Extended GCD algorithm
        # http://en.wikipedia.org/wiki/Polynomial_greatest_common_divisor

        if self._value == 0:
            raise ValueError("Inversion of zero")

        r0, r1 = self._value, self.irr_poly
        s0, s1 = 1, 0
        while r1 > 0:
            q = _div_gf2(r0, r1)[0]
            r0, r1 = r1, r0 ^ _mult_gf2(q, r1)
            s0, s1 = s1, s0 ^ _mult_gf2(q, s1)
        return _Element(s0)

    def __pow__(self, exponent):
        result = _Element(self._value)
        for _ in range(exponent - 1):
            result = result * self
        return result

class Shamir(object):
    """Shamir's secret sharing scheme.

    A secret is split into ``n`` shares, and it is sufficient to collect
    ``k`` of them to reconstruct the secret.
    """

    @staticmethod
    def split(k, n, secret):
        """Split a secret into ``n`` shares.

        The secret can be reconstructed later using just ``k`` shares
        out of the original ``n``.
        Each share must be kept confidential to the person it was
        assigned to.

        Each share is associated to an index (starting from 1).

        Args:
          k (integer):
            The sufficient number of shares to reconstruct the secret (``k < n``).
          n (integer):
            The number of shares that this method will create.
          secret (byte string):
            A byte string of 16 bytes (e.g. the AES 128 key).

        Return (tuples):
            ``n`` tuples. A tuple is meant for each participant and it contains two items:

            1. the unique index (an integer)
            2. the share (a byte string, 16 bytes)
        """

        #
        # We create a polynomial with random coefficients in GF(2^128):
        #
        # p(x) = \sum_{i=0}^{k-1} c_i * x^i
        #
        # c_0 is the encoded secret
        #

        coeffs = [_Element(rng(16)) for i in range(k - 1)]
        coeffs.append(_Element(secret))

        # Each share is y_i = p(x_i) where x_i is the public index
        # associated to each of the n users.

        def make_share(user, coeffs):
            idx = _Element(user)
            share = _Element(0)
            for coeff in coeffs:
                share = idx * share + coeff

            return share.encode()

        return [(i, make_share(i, coeffs)) for i in range(1, n + 1)]

    @staticmethod
    def combine(shares):
        """Recombine a secret, if enough shares are presented.

        Args:
          shares (tuples):
            The *k* tuples, each containin the index (an integer) and
            the share (a byte string, 16 bytes long) that were assigned to
            a participant.
          ssss (bool):
            If ``True``, the shares were produced by the ``ssss`` utility.
            Default: ``False``.

        Return:
            The original secret, as a byte string (16 bytes long).
        """

        k = len(shares)

        gf_shares = []
        for x in shares:
            idx = _Element(x[0])
            value = _Element(x[1])
            if any(y[0] == idx for y in gf_shares):
                raise ValueError("Duplicate share")
            gf_shares.append((idx, value))

        result = _Element(0)
        for j in range(k):
            x_j, y_j = gf_shares[j]

            numerator = _Element(1)
            denominator = _Element(1)

            for m in range(k):
                x_m = gf_shares[m][0]
                if m != j:
                    numerator *= x_m
                    denominator *= x_j + x_m
            result += y_j * numerator * denominator.inverse()
        return result.encode()
\end{lstlisting}

\newpage
widget.py
\begin{lstlisting}[language = Python]
# This Python file uses the following encoding: utf-8
import os
from pathlib import Path
import sys

from shamir import Shamir
from binascii import hexlify, unhexlify

from PySide2.QtWidgets import QApplication, QWidget
from PySide2.QtCore import QFile
from PySide2.QtUiTools import QUiLoader

class Widget(QWidget):
    def __init__(self):
        super(Widget, self).__init__()
        self.ui = QUiLoader().load('form.ui')
        self.ui.show()
        self.ui.pushButton_create_share.clicked.connect(self.create_shares)
        self.ui.pushButton_combine_share.clicked.connect(self.combine_shares)

    def create_shares(self):
        secret = self.ui.plainTextEdit_Secret.toPlainText().encode()
        l = Shamir.split(3, 5, secret)
        print(l)
        self.ui.plainTextEdit_s1.setPlainText(str(hexlify(l[0][1]))[2:-1])
        self.ui.plainTextEdit_s2.setPlainText(str(hexlify(l[1][1]))[2:-1])
        self.ui.plainTextEdit_s3.setPlainText(str(hexlify(l[2][1]))[2:-1])
        self.ui.plainTextEdit_s4.setPlainText(str(hexlify(l[3][1]))[2:-1])
        self.ui.plainTextEdit_s5.setPlainText(str(hexlify(l[4][1]))[2:-1])
        pass

    def combine_shares(self):
        self.ui.plainTextEdit_cs.setPlainText("NULL")
        shares = []
        if len(self.ui.plainTextEdit_s1.toPlainText()) != 0:
            shares.append((1, unhexlify(self.ui.plainTextEdit_s1.toPlainText().encode())))
        if len(self.ui.plainTextEdit_s2.toPlainText()) != 0:
            shares.append((2, unhexlify(self.ui.plainTextEdit_s2.toPlainText().encode())))
        if len(self.ui.plainTextEdit_s3.toPlainText()) != 0:
            shares.append((3, unhexlify(self.ui.plainTextEdit_s3.toPlainText().encode())))
        if len(self.ui.plainTextEdit_s4.toPlainText()) != 0:
            shares.append((4, unhexlify(self.ui.plainTextEdit_s4.toPlainText().encode())))
        if len(self.ui.plainTextEdit_s5.toPlainText()) != 0:
            shares.append((5, unhexlify(self.ui.plainTextEdit_s5.toPlainText().encode())))
        print(shares)
        secret = Shamir.combine(shares)
        self.ui.plainTextEdit_cs.setPlainText(secret.decode())
        pass

if __name__ == "__main__":
    app = QApplication([])
    widget = Widget()
    widget.show()
    sys.exit(app.exec_())
\end{lstlisting}

\newpage
form.ui
\begin{lstlisting}[language = html]
<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0">
 <class>Widget</class>
 <widget class="QWidget" name="Widget">
  <property name="geometry">
   <rect>
    <x>0</x>
    <y>0</y>
    <width>557</width>
    <height>313</height>
   </rect>
  </property>
  <property name="windowTitle">
   <string>Widget</string>
  </property>
  <layout class="QGridLayout" name="gridLayout">
   <item row="0" column="0" colspan="3">
    <widget class="QLabel" name="label">
     <property name="text">
      <string>3, 5 - Shamir Secret Sharing</string>
     </property>
    </widget>
   </item>
   <item row="1" column="0">
    <widget class="QLabel" name="label_2">
     <property name="text">
      <string>Secret</string>
     </property>
    </widget>
   </item>
   <item row="1" column="1">
    <widget class="QLabel" name="label_3">
     <property name="text">
      <string>Share 1</string>
     </property>
    </widget>
   </item>
   <item row="1" column="2">
    <widget class="QLabel" name="label_4">
     <property name="text">
      <string>Share 2</string>
     </property>
    </widget>
   </item>
   <item row="1" column="3">
    <widget class="QLabel" name="label_8">
     <property name="text">
      <string>Combined Share</string>
     </property>
    </widget>
   </item>
   <item row="2" column="0">
    <widget class="QPlainTextEdit" name="plainTextEdit_Secret"/>
   </item>
   <item row="2" column="1">
    <widget class="QPlainTextEdit" name="plainTextEdit_s1"/>
   </item>
   <item row="2" column="2">
    <widget class="QPlainTextEdit" name="plainTextEdit_s2"/>
   </item>
   <item row="2" column="3" rowspan="3">
    <widget class="QPlainTextEdit" name="plainTextEdit_cs"/>
   </item>
   <item row="3" column="0">
    <widget class="QLabel" name="label_5">
     <property name="text">
      <string>Share 3</string>
     </property>
    </widget>
   </item>
   <item row="3" column="1">
    <widget class="QLabel" name="label_6">
     <property name="text">
      <string>Share 4</string>
     </property>
    </widget>
   </item>
   <item row="3" column="2">
    <widget class="QLabel" name="label_7">
     <property name="text">
      <string>Share 5</string>
     </property>
    </widget>
   </item>
   <item row="4" column="0">
    <widget class="QPlainTextEdit" name="plainTextEdit_s3"/>
   </item>
   <item row="4" column="1">
    <widget class="QPlainTextEdit" name="plainTextEdit_s4"/>
   </item>
   <item row="4" column="2">
    <widget class="QPlainTextEdit" name="plainTextEdit_s5"/>
   </item>
   <item row="5" column="0">
    <widget class="QPushButton" name="pushButton_create_share">
     <property name="text">
      <string>Create Shares</string>
     </property>
    </widget>
   </item>
   <item row="5" column="1">
    <widget class="QPushButton" name="pushButton_combine_share">
     <property name="text">
      <string>Combine Shares</string>
     </property>
    </widget>
   </item>
  </layout>
 </widget>
 <resources/>
 <connections/>
</ui>

\end{lstlisting}

\newpage
image.py
\begin{lstlisting}[language = Python]
from statistics import mode
import numpy as np
import matplotlib.pyplot as plt
from shamir import *
from binascii import hexlify

img = plt.imread('cat.jpg')
# plt.imshow(img)

def img_split(k, n, img: np.ndarray):
    a, b, c = img.shape
    l = []
    for i in range(n):
        l.append(np.zeros((a, b, c), np.dtype('int')))
    for i in range(a):
        for j in range(b):
            for kk in range(c):
                tmp_share = Shamir.split(k, n, int(img[i, j, kk]))
                for idx in range(n):
                    l[idx][i, j, kk] = bytes_to_long(tmp_share[idx][1])
    return l

tmp_l = img_split(3, 5, img)
tmp_l.append(img)

print('imhere')

def show_images(images: list[np.ndarray]) -> None:
    n: int = len(images)
    f = plt.figure()
    for i in range(n):
        # Debug, plot figure
        f.add_subplot(1, n, i + 1)
        plt.imshow(images[i])

    plt.show(block=True)

show_images(tmp_l)

plt.show()
\end{lstlisting}

\newpage
\section{Many Time Pad}
\label{appendix:mtp}
main.py
\begin{lstlisting}[language = Python]
from binascii import unhexlify
from string import ascii_letters
from math import sqrt
import utils

def main():
    cipher_texts = []
    with open('cipher_text.txt', encoding='utf-8') as file:
        content = file.read().split('\n')
    for i in range(1, 21, 2):
        cipher_texts.append(unhexlify(content[i]))

    with open('dest_text.txt', encoding='utf-8') as file:
        content = file.read()
    dest_text = unhexlify(content)

    letters = ascii_letters.encode('ascii')     ## ASCII letters

    key = [0] * 1024

    for ct_a in cipher_texts:
        possible_pos = [0] * len(ct_a)
        for ct_b in cipher_texts:
            if ct_a == ct_b:
                continue
            guess_str = utils.byte_xor(ct_a, ct_b)
            for idx, guess_char in enumerate(guess_str):
                if guess_char not in letters and guess_char != 0:
                    continue
                possible_pos[idx] += 1

        threshold = len(cipher_texts) - sqrt(len(cipher_texts))   ## Magic threshold

        for i in range(len(ct_a)):
            if possible_pos[i] > threshold:
                key[i] = ct_a[i] ^ 0x20

    print(utils.byte_xor(dest_text, key))

if __name__ == '__main__':
    main()
\end{lstlisting}

\newpage
utils.py
\begin{lstlisting}[language = Python]
def byte_xor(a: bytes, b: bytes) -> bytes:
    '''
    :return: xor result of a and b
    '''
    if len(a) > len(b):
        return bytes([x ^ y for x, y in zip(a[:len(b)], b)])
    else:
        return bytes([x ^ y for x, y in zip(a, b[:len(a)])])
\end{lstlisting}

\newpage
\section{AES}
\label{appendix:aes}
main.py
\begin{lstlisting}[language = Python]
from binascii import unhexlify
from utils import *

def main():
    keys = []
    cipher_texts = []
    with open('infos.txt', encoding='utf-8') as file:
        content = file.read().split('\n')
    for i in range(1, 16, 4):
        keys.append(unhexlify(content[i]))
        cipher_texts.append(unhexlify(content[i+2]))

    print(CBC.decrypt(cipher_texts[0], keys[0]))
    print(CBC.decrypt(cipher_texts[1], keys[1]))
    print(CTR.decrypt(cipher_texts[2], keys[2]))
    print(CTR.decrypt(cipher_texts[3], keys[3]))

if __name__ == '__main__':
    main()
\end{lstlisting}

utils.py
\begin{lstlisting}[language = Python]
import binascii
from Crypto.Cipher import AES
from cxc_toolkit import integer

def byte_xor(a: bytes, b: bytes) -> bytes:
    '''
    :return: xor result of a and b
    '''
    if len(a) > len(b):
        return bytes([x ^ y for x, y in zip(a[:len(b)], b)])
    else:
        return bytes([x ^ y for x, y in zip(a, b[:len(a)])])

def to_int(byte):
    """
    Convert bytes to int

    :type byte: bytes
    :rtype: int
    """
    s = 0
    for i, number in enumerate(byte):
        s = s * 256 + number
    return s

def byte_add(byte, addtions):
    """
    Add int to bytes

    :type byte: bytes
    :type addtions: int
    :rtype: bytes
    """
    return integer.to_bytes(to_int(byte) + addtions, bytes_size=len(byte))

def msg_block_generator(msg, padding=False):
        while len(msg) >= 16:
            yield msg[:16]
            msg = msg[16:]
        if len(msg) > 0:
            if not padding:
                yield msg
                return

            reminder = 16 - len(msg)
            msg = msg + bytes([reminder]) * reminder
            yield msg
        else:
            yield b'16' * 16

def cipher_block_generator(cipher):
    while len(cipher):
        yield cipher[:16]
        cipher = cipher[16:]

class CBC:

    def encrypt(msg, key, iv):
        cipher = AES.new(key, AES.MODE_ECB)
        cipher_block = iv
        ciphertext = iv
        for msg_block in msg_block_generator(msg, padding=True):
            cipher_block = cipher.encrypt(byte_xor(cipher_block, msg_block))
            ciphertext += cipher_block
        return ciphertext

    def decrypt(cipher_text, key):
        cipher = AES.new(key, AES.MODE_ECB)
        iv, cipher_text = cipher_text[:16], cipher_text[16:]
        msg = b''
        for cipher_block in cipher_block_generator(cipher_text):
            msg_block = byte_xor(cipher.decrypt(cipher_block), iv)
            iv = cipher_block
            msg += msg_block
        if msg[-16:] == b'\x16' * 16:
            return msg[:-16]
        pad_bytes = msg[-1]
        reminder = len(msg) - pad_bytes
        if msg[reminder:] == bytes([pad_bytes]) * pad_bytes:
            return msg[:reminder]
        else:
            print('Cipher text is invalid')

class CTR:

    def encrypt(msg, key, iv):
        cipher = AES.new(key, AES.MODE_ECB)
        ciphertext = b''
        for i, msg_block in enumerate(msg_block_generator(msg, padding=False)):
            cipher_block = cipher.encrypt(byte_add(iv, i))
            cipher_block = byte_xor(msg_block, cipher_block)
            ciphertext += cipher_block
        return ciphertext

    def decrypt(cipher_text, key):
        iv, cipher_text = cipher_text[:16], cipher_text[16:]
        cipher = AES.new(key, AES.MODE_ECB)
        msg = b''
        for i, cipher_block in enumerate(cipher_block_generator(cipher_text)):
            iv_encrypted = cipher.encrypt(byte_add(iv, i))
            msg_block = byte_xor(cipher_block, iv_encrypted)
            msg += msg_block
        return msg

\end{lstlisting}


\chapter{CT Block Diagrams}

\section{The Four Basic Motifs}

Understanding complex systems, with many interconnections, is aided by graphical representations, generally called block diagrams \footnote{There is a closely related graphical approach called \emph{signal flow graphs} that you may learn about in upper-level courses. They are equivalent to block diagrams, but are more amenable to computer representation and manipulation.}. They are a hybrid graphical-analytical approach.

There are just four basic motifs needed to build any block diagram. Let $\mathcal{S}_i$ denote a (sub) system. Then the four motifs are:

\begin{itemize}
\item A single block.\\[1em]
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system) {$\mathcal{S}_1$};
    \node [output, right of=system] (output) {};

    % Once the nodes are placed, connecting them is easy.
    \draw [draw,->] (input) -- node {$x(t)$} (system);
    \draw [->] (system) -- node {$y(t)$} (output);
\end{tikzpicture}

\item A {\it series} connection of two blocks\\[1em]
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system1) {$\mathcal{S}_1$};
    \node [block, right of=system1,node distance=4cm] (system2) {$\mathcal{S}_2$};
    \node [output, right of=system2] (output) {};

    % Once the nodes are placed, connecting them is easy.
    \draw [draw,->] (input) -- node {$x(t)$} (system1);
    \draw [->] (system1) -- (system2);
    \draw [->] (system2) -- node {$y(t)$} (output);
\end{tikzpicture}
\item A {\it parallel} connection of two blocks\\[1em]
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks and inputs
    \node[shape=coordinate] at (1,1) (input1) {};
    \node[block] at (3,1) (block1) {$\mathcal{S}_1$};
    \node[shape=coordinate] at ($(block1.east)+(0.5,0)$) (output1) {};
    \draw[->] (input1) -- (block1);
    \draw (block1) -- (output1);

    \node[shape=coordinate] at (1,-1) (input2) {};
    \node[block] at (3,-1) (block2) {$\mathcal{S}_2$};
    \node[shape=coordinate] at ($(block2.east)+(0.5,0)$) (output2) {};
    \draw[->] (input2) -- (block2);
    \draw (block2) -- (output2);

    \node [input, name=input] at (0,0) {};
    \node [input, name=conn] at (1,0) {};
    \draw (conn) -- (input1);
    \draw (conn) -- (input2);
    \node [sum, right of=input,node distance=5cm] (sum) {$\Sigma$};
    \draw [->] (output1) -| (sum);
    \draw [->] (output2) -| (sum);

    \draw [draw] (input) -- node {$x(t)$} (conn);
    \node [output, right of=sum] (output) {};
    \draw [->] (sum) -- node {$y(t)$} (output);
\end{tikzpicture}

\item A {\it feedback} connection\\[1em]
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node[block] at (4,0) (block1) {$\mathcal{S}_1$};

    \node[block] at (4,-2) (block2) {$\mathcal{S}_2$};
    \node[shape=coordinate] at (6,-2) (input2) {};

    \node [input, name=input] at (0,0) {};
    \node [shape=coordinate, name=conn] at (6,0) {};
    \draw (block1) -- (conn);
    \draw (conn) -- (input2);
    \draw [->] (input2) -- (block2);

    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};
    \draw [->] (block2) -| node[pos=0.95] {$-$} (sum);

    \draw [draw,->] (input) -- node {$x(t)$} (sum);
    \draw [->] (sum) -- (block1);
    \node [output, right of=conn] (output) {};
    \draw [->] (conn) -- node {$y(t)$} (output);
\end{tikzpicture}
\end{itemize}

Note the feedback is negative (the minus sign on the feedback summation input). These can be use in various combinations, as we shall see shortly.

\section{Connections to Convolution}

Each subsystem, $\mathcal{S}_i$, can be represented by a basic time-domain operation (e.g. derivatives, integrals, addition, and scaling) or more generally by it's impulse response $h_i(t)$.
\
For example a block representing an system acting as integrator is typically drawn as

\begin{center}
  \begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system) {$\int$};
    \node [output, right of=system] (output) {};

    % Once the nodes are placed, connecting them is easy.
    \draw [draw,->] (input) -- node {$x(t)$} (system);
    \draw [->] (system) -- node[pos=3] {$y(t) = \int\limits_{-\infty}^t x(\tau) \; d\tau$} (output);
\end{tikzpicture}
\end{center}
This is equivalent to an impulse response $h(t) = u(t)$ so that it might also be drawn as
\begin{center}
  \begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system) {$h(t) = u(t)$};
    \node [output, right of=system] (output) {};

    % Once the nodes are placed, connecting them is easy.
    \draw [draw,->] (input) -- node {$x(t)$} (system);
    \draw [->] (system) -- node[pos=3] {$y(t) = x(t) * u(t) = \int\limits_{-\infty}^t x(\tau) \; d\tau$} (output);
\end{tikzpicture}
\end{center}

We can use the concept of convolution to connect block diagrams to the properties of convolution

\begin{itemize}
\item A single block is equivalent to convolution with the impulse response for that subsystem\\[1em]
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system) {$h_1(t)$};
    \node [output, right of=system] (output) {};

    % Once the nodes are placed, connecting them is easy.
    \draw [draw,->] (input) -- node {$x(t)$} (system);
    \draw [->] (system) -- node[pos=2] {$y(t) = h_1(t)*x(t)$} (output);
\end{tikzpicture}

\item Using the associative property, a series connection of two blocks becomes
  \begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node [input, name=input] {};
    \node [block, right of=input] (system1) {$h_1(t)$};
    \node [block, right of=system1,node distance=4cm] (system2) {$h_2(t)$};
    \node [output, right of=system2] (output) {};

    \draw [draw,->] (input) -- node {$x(t)$} (system1);
    \draw [->] (system1) -- (system2);
    \draw [->] (system2) -- node[pos=3] {$y(t) = \left[h_1(t)*h_2(t)\right]*x(t)$} (output);
\end{tikzpicture}
  \end{center}
  which can be reduced to a single convolution $y(t) = h_3(t)*x(t)$ where $h_3(t) = h_1(t)*h_2(t)$.
\item Using the distributive property, a parallel connection of two blocks becomes
  \begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]

    \node[shape=coordinate] at (1,1) (input1) {};
    \node[block] at (3,1) (block1) {$h_1(t)$};
    \node[shape=coordinate] at ($(block1.east)+(0.5,0)$) (output1) {};
    \draw[->] (input1) -- (block1);
    \draw (block1) -- (output1);

    \node[shape=coordinate] at (1,-1) (input2) {};
    \node[block] at (3,-1) (block2) {$h_2(t)$};
    \node[shape=coordinate] at ($(block2.east)+(0.5,0)$) (output2) {};
    \draw[->] (input2) -- (block2);
    \draw (block2) -- (output2);

    \node [input, name=input] at (0,0) {};
    \node [input, name=conn] at (1,0) {};
    \draw (conn) -- (input1);
    \draw (conn) -- (input2);
    \node [sum, right of=input,node distance=5cm] (sum) {$\Sigma$};
    \draw [->] (output1) -| (sum);
    \draw [->] (output2) -| (sum);

    \draw [draw] (input) -- node {$x(t)$} (conn);
    \node [output, right of=sum] (output) {};
    \draw [->] (sum) -- node[pos=3] {$y(t)= \left[h_1(t)*x(t)\right] +  \left[h_2(t)*x(t)\right] =  \left[h_1(t)+h_2(t)\right]*x(t)$} (output);
\end{tikzpicture}
  \end{center}
  which is equivalent to a single convolution $y(t) = h_3(t)*x(t)$ where $h_3(t) = h_1(t) + h_2(t)$.
\item In the feedback connection let $w(t)$ be the output of the summation
  \begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    % We start by placing the blocks
    \node[block] at (4.5,0) (block1) {$h_1(t)$};

    \node[block] at (4,-2) (block2) {$h_2(t)$};
    \node[shape=coordinate] at (6,-2) (input2) {};

    \node [input, name=input] at (0,0) {};
    \node [shape=coordinate, name=conn] at (6,0) {};
    \draw (block1) -- (conn);
    \draw (conn) -- (input2);
    \draw [->] (input2) -- (block2);

    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};
    \draw [->] (block2) -| node[pos=0.95] {$-$} (sum);

    \draw [draw,->] (input) -- node {$x(t)$} (sum);
    \draw [->] (sum) -- (block1);
    \node [output, right of=conn] (output) {};
    \draw [->] (conn) -- node {$y(t)$} (output);
    \draw node at (3,0.3) {$w(t)$};
\end{tikzpicture}
  \end{center}
  Then $y(t) = h_1(t)*w(t)$ and $w(t) = x(t) - h_2(t)*y(t)$. Substituting the later into the former gives $y(t) = h_1*(x-h_2(t)*y(t))$. Using the distributive property we get $y(t) = h_1(t)*x(t) - h_1(t)*h_2(t)*y(t)$. Isolating the input on the right-hand side and using $y(t) = \delta(t)*y(t)$ we get
  \[
  y(t) + h_1(t)*h_2(t)*y(t) = \left[\delta(t) + h_1(t)*h_2(t)\right]*y(t) = h_1(t)*x(t)
  \]
  We can solve this for $y(t)$ using the concept of inverse systems. Let $h_3(t)* \left[\delta(t) + h_1(t)*h_2(t)\right]= \delta(t)$, i.e. $h_3$ is the inverse system of $\delta(t) + h_1(t)*h_2(t)$. Then
  \[
  y(t) = h_3(t)*h_1(t)*x(t)
  \]
\end{itemize}

Recall, when the system is instantaneous (memoryless) the impulse response is $a\delta(t)$ for some constant $a$. This is the same as scaling the signal by $a$. We typically drop the block in such cases and draw the input-output operation as

\begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
  \node [input, name=input] at (0,0) {};
  \node [output, name=system] at (2,0) {};
  \node [output, name=output] at (4,0) {};
  \draw [draw,->] (input) -- node {$x(t)$} (system);
  \draw [draw,->] (system) -- node[pos=1] {$y(t) = ax(t)$} (output);
  \draw [->] (input) -- node {$a$} (output);
\end{tikzpicture}
\end{center}

These properties allow us to perform transformations, either breaking up a system into subsystems, or reducing a system to a single block.

\begin{example}
  Consider a second-order system system with impulse response
  \[
  h(t) = \left(e^{-3t} - e^{-t}\right)\, u(t)
  \]
  We can express this as a block diagram consisting of two parallel blocks
  \begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]

    \node[shape=coordinate] at (1,1) (input1) {};
    \node[block] at (3,1) (block1) {$h_1(t) = e^{-3t}u(t)$};
    \node[shape=coordinate] at ($(block1.east)+(0.5,0)$) (output1) {};
    \draw[->] (input1) -- (block1);
    \draw (block1) -- (output1);

    \node[shape=coordinate] at (1,-1) (input2) {};
    \node[block] at (3,-1) (block2) {$h_2(t) = -e^{-t}u(t)$};
    \node[shape=coordinate] at ($(block2.east)+(0.5,0)$) (output2) {};
    \draw[->] (input2) -- (block2);
    \draw (block2) -- (output2);

    \node [input, name=input] at (0,0) {};
    \node [input, name=conn] at (1,0) {};
    \draw (conn) -- (input1);
    \draw (conn) -- (input2);
    \node [sum, right of=input,node distance=5cm] (sum) {$\Sigma$};
    \draw [->] (output1) -| (sum);
    \draw [->] (output2) -| (sum);

    \draw [draw] (input) -- node {$x(t)$} (conn);
    \node [output, right of=sum] (output) {};
    \draw [->] (sum) -- node[pos=1] {$y(t)$} (output);
\end{tikzpicture}
  \end{center}

\end{example}

\begin{example}
  Consider a system with block diagram

  \begin{center}
\begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]

    \node[shape=coordinate] at (1,1) (input1) {};
    \node[block] at (3,1) (block1) {$h_1(t) = e^{-2t}u(t)$};
    \node[shape=coordinate] at ($(block1.east)+(0.5,0)$) (output1) {};
    \draw[->] (input1) -- (block1);
    \draw (block1) -- (output1);

    \node[shape=coordinate] at (1,-1) (input2) {};
    \node[block] at (3,-1) (block2) {$h_2(t) = -e^{-4t}u(t)$};
    \node[shape=coordinate] at ($(block2.east)+(0.5,0)$) (output2) {};
    \draw[->] (input2) -- (block2);
    \draw (block2) -- (output2);

    \node[block] at (8,0) (block3) {$h_3(t) = e^{-6t}u(t)$};

    \node [input, name=input] at (0,0) {};
    \node [input, name=conn] at (1,0) {};
    \draw (conn) -- (input1);
    \draw (conn) -- (input2);
    \node [sum, right of=input,node distance=5cm] (sum) {$\Sigma$};
    \draw [->] (output1) -| (sum);
    \draw [->] (output2) -| (sum);

    \draw [draw] (input) -- node {$x(t)$} (conn);
    \node [output, right of=block3] (output) {};
    \draw [->] (sum) -- (block3);
    \draw [->] (block3) -- node[pos=1] {$y(t)$} (output);
\end{tikzpicture}
  \end{center}
  We can determine the overall impulse response of this system using the distributive and associative properties
\begin{align*}
  h(t) &= \left[ h_1(t) + h_2(t)\right]*h_3(t)\\
  &= h_1(t)*h_3(t) + h_2(t)*h_3(t)\\
  &= \left[ e^{-2t}u(t)\right]*\left[ e^{-6t}u(t)\right] + \left[-e^{-4t}u(t) \right]*\left[ e^{-6t}u(t)\right]
\end{align*}
Using the convolution table from Lecture 8 we get the overall impulse response
\[
h(t) = \frac{e^{-2 t}-e^{-6 t}}{4}u(t) - \frac{e^{-4 t}-e^{-6 t}}{2}u(t) = \frac{1}{4}e^{-2t}u(t) -\frac{1}{2}e^{-4t}u(t) + \frac{1}{4}e^{-6t}u(t)
\]
\end{example}

\section{Connections to LCCDE}

The other system representation we have seen are linear, constant-coefficient differential equations. These can be expressed as combinations of derivative and/or integration blocks.

\subsection*{First-Order System}

To illustrate this consider the first-order LCCDE
\[
\frac{dy}{dt}(t) + ay(t) = x(t)
\]
We can solve this for $y(t)$
\[
y(t) = -\frac{1}{a} \frac{dy}{dt}(t) + \frac{1}{a}x(t)
\]
and can express this as a feedback motif
\begin{center}
  \begin{tikzpicture}[auto]
    \node[block] at (4,-2) (block2) {$\frac{1}{a}\frac{d}{dt}$};
    \node[shape=coordinate] at (6,-2) (input2) {};
    \node [input, name=input] at (0,0) {};
    \node [shape=coordinate, name=conn] at (6,0) {};
    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};

    \draw (sum) -- (conn);
    \draw (conn) -- (input2);
    \draw [->] (input2) -- (block2);
    \draw [->] (block2) -| node[pos=0.95] {$-$} (sum);
    \draw [draw,->] (input) -- node {$\frac{1}{a}$} (sum);
    \node [left of=input, node distance=2em] {$x(t)$};
    \node [output, right of=conn] (output) {};
    \draw [->] (conn) -- (output);
    \node [right of=output, node distance=2em] {$y(t)$};
\end{tikzpicture}
\end{center}

Alternatively we could integrate the differential equation
\begin{align*}
  \frac{dy}{dt}(t) + ay(t) &= x(t)\\
  \int\limits_{-\infty}^t \frac{dy}{dt}(\tau)\; d\tau + a\int\limits_{-\infty}^t y(\tau)\; d\tau &= \int\limits_{-\infty}^t x(\tau)\; d\tau\\
  y(\tau) \Big|_{-\infty}^t  + a\int\limits_{-\infty}^t y(\tau)\; d\tau &= \int\limits_{-\infty}^t x(\tau)\; d\tau\\
\end{align*}
Under the assumption $y(-\infty) = 0$ we can solve this for $y(t)$ to get
\[
  y(t) = -a\int\limits_{-\infty}^t y(\tau)\; d\tau + \int\limits_{-\infty}^t x(\tau)\; d\tau
\]
which can be expressed as the block diagram
\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node[block] at (2,0) (block1) {$\int$};
    \node[block] at (6,-1) (block2) {$\int$};
    \node[shape=coordinate] at (6,-2) (input2) {};

    \node [shape=coordinate, name=conn] at (6,0) {};
    \node [shape=coordinate, name=conn2] at (4,-2) {};
    \node [shape=coordinate, name=conn3] at (6,-2) {};
    \node [sum, right of=block1,node distance=2cm] (sum) {$\Sigma$};
    \node [output, right of=conn] (output) {};

    \draw (sum) -- (conn);
    \draw (conn) -- (block2);
    \draw (block2) -- (conn3);
    \draw (conn3) -- node {$a$} (conn2);
    \draw [->] (conn2) -| node[pos=0.95] {$-$} (sum);
    \draw [draw,->] (input) -- node {$x(t)$} (block1);
    \draw [->] (block1) -- (sum);
    \draw [->] (conn) -- node {$y(t)$} (output);
  \end{tikzpicture}
\end{center}

We can simplify this block diagram, by noting
\begin{align*}
  y(t) &= -a\int\limits_{-\infty}^t y(\tau)\; d\tau + \int\limits_{-\infty}^t
  x(\tau)\; d\tau\\
  &= \int\limits_{-\infty}^t \left(-a y(\tau) +  x(\tau)\right)\; d\tau\\
\end{align*}
which requires only a single integrator
\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node[block] at (4,-1) (block2) {$\int$};

    \node [shape=coordinate, name=conn] at (4,0) {};
    \node [shape=coordinate, name=conn2] at (2,-2) {};
    \node [shape=coordinate, name=conn3] at (4,-2) {};
    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};
    \node [output, right of=conn3] (output) {};

    \draw (sum) -- (conn);
    \draw (conn) -- (block2);
    \draw (block2) -- (conn3);
    \draw (conn3) -- node {$a$} (conn2);
    \draw [->] (conn2) -| node[pos=0.95] {$-$} (sum);
    \draw [draw,->] (input) -- node {$x(t)$} (sum);
    \draw [->] (conn3) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}
\end{center}

The choice of using derivative or integrator blocks is not arbitrary in practice. Derivatives are sensitive to noise at high frequencies (for reasons we will see later in the semester) and so integrators perform much better when implemented in hardware.

\subsection*{Second-Order System}

Now consider the second-order system
\[
\frac{d^2y}{dt^2}(t) + a\frac{dy}{dt}(t)  + by(t)= x(t)
\]
Using a similar process to the first-order system, we can express this as (dropping the limits of integration for clarity):
\[
y(t) = -a \int y(\tau)\; d\tau + \int\int \left( -by(\tau) + x(\tau) \right) \; d\tau^2
\]
which has the block diagram

\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [block, right of=input,node distance=2cm] (block1) {$\int$};
    \node [block, right of=block1,node distance=2cm] (block2) {$\int$};
    \node [sum, right of=block2,node distance=2cm] (sum) {$\Sigma$};
    \node [sum, below of=sum,node distance=2cm] (sum2) {$\Sigma$};
    \node[block] at (8,-1) (block3) {$\int$};
    \node[block] at (8,-3) (block4) {$\int$};

    \node [shape=coordinate, name=conn1] at (8,0) {};
    \node [shape=coordinate, name=conn2] at (8,-2) {};
    \node [shape=coordinate, name=conn3] at (8,-4) {};
    \node [shape=coordinate, name=conn4] at (6,-4) {};
    \node [output, right of=conn1] (output) {};

    \draw [->] (input) -- node {$x(t)$} (block1);
    \draw [->] (block1) -- (block2);
    \draw [->] (block2) -- (sum);
    \draw (sum) -- (conn1);
    \draw [->] (conn1) -- (block3);
    \draw (block3) -- (conn2);
    \draw [->] (conn2) -- (block4);
    \draw [->] (conn2) -- node {$a$} (sum2);
    \draw (block4) -- (conn3);
    \draw (conn3) -- node {$b$} (conn4);
    \draw [->] (conn3) -| (sum2);
    \draw [->] (sum2) -- node[pos=0.95] {$-$} (sum);
    \draw [->] (conn1) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}
\end{center}
This is equivalent to two systems in series

\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input1] at (0,0) {};
    \node [block, right of=input1,node distance=2cm] (block1) {$\int$};
    \node [block, right of=block1,node distance=2cm] (block2) {$\int$};
    \node [output, right of=block2] (output) {};

    \draw [->] (input1) -- node {$x(t)$} (block1);
    \draw [->] (block1) -- (block2);
    \draw [->] (block2) -- node[pos=1] {$z(t)$} (output);

    \node [input, name=input2] at (6,0) {};
    \node [sum, right of=input2,node distance=2cm] (sum) {$\Sigma$};
    \node [sum, below of=sum,node distance=2cm] (sum2) {$\Sigma$};
    \node[block] at (10,-1) (block3) {$\int$};
    \node[block] at (10,-3) (block4) {$\int$};

    \node [shape=coordinate, name=conn1] at (10,0) {};
    \node [shape=coordinate, name=conn2] at (10,-2) {};
    \node [shape=coordinate, name=conn3] at (10,-4) {};
    \node [shape=coordinate, name=conn4] at (8,-4) {};
    \node [output, right of=conn1] (output) {};

    \draw [->] (input2) -- node {$z(t)$} (sum);
    \draw (sum) -- (conn1);
    \draw [->] (conn1) -- (block3);
    \draw (block3) -- (conn2);
    \draw [->] (conn2) -- (block4);
    \draw [->] (conn2) -- node {$a$} (sum2);
    \draw (block4) -- (conn3);
    \draw (conn3) -- node {$b$} (conn4);
    \draw [->] (conn3) -| (sum2);
    \draw [->] (sum2) -- node[pos=0.95] {$-$} (sum);
    \draw [->] (conn1) -- node[pos=1] {$y(t)$} (output);
    \end{tikzpicture}
\end{center}

Recall that, from the commutative property of convolution, the order of systems in series can be swapped\\

\begin{center}
\tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};
    \node [sum, below of=sum,node distance=2cm] (sum2) {$\Sigma$};
    \node[block] at (4,-1) (block3) {$\int$};
    \node[block] at (4,-3) (block4) {$\int$};

    \node [shape=coordinate, name=conn1] at (4,0) {};
    \node [shape=coordinate, name=conn2] at (4,-2) {};
    \node [shape=coordinate, name=conn3] at (4,-4) {};
    \node [shape=coordinate, name=conn4] at (2,-4) {};
    \node [output, right of=conn1] (output) {};

    \draw [->] (input) -- node {$x(t)$} (sum);
    \draw (sum) -- (conn1);
    \draw [->] (conn1) -- (block3);
    \draw (block3) -- (conn2);
    \draw [->] (conn2) -- (block4);
    \draw [->] (conn2) -- node {$a$} (sum2);
    \draw (block4) -- (conn3);
    \draw (conn3) -- node {$b$} (conn4);
    \draw [->] (conn3) -| (sum2);
    \draw [->] (sum2) -- node[pos=0.95] {$-$} (sum);
    \draw [->] (conn1) -- node[pos=1] {$z(t)$} (output);

    \node [input, name=input] at (6,0) {};
    \node [block, right of=input,node distance=2cm] (block1) {$\int$};
    \node [block, right of=block1,node distance=2cm] (block2) {$\int$};
    \node [output, right of=block2] (output) {};

    \draw [->] (input) -- node {$z(t)$} (block1);
    \draw [->] (block1) -- (block2);
    \draw [->] (block2) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}
\end{center}
We then note that the signal $z$ and the output of the integrator blocks are the same in both systems so that they can be combined into a single block diagram as follows, reducing the number of integrators by two

\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [sum, right of=input,node distance=2cm] (sum) {$\Sigma$};
    \node [sum, below of=sum,node distance=2cm] (sum2) {$\Sigma$};
    \node[block] at (4,-1) (block3) {$\int$};
    \node[block] at (4,-3) (block4) {$\int$};

    \node [shape=coordinate, name=conn1] at (4,0) {};
    \node [shape=coordinate, name=conn2] at (4,-2) {};
    \node [shape=coordinate, name=conn3] at (4,-4) {};
    \node [shape=coordinate, name=conn4] at (2,-4) {};
    \node [output, right of=conn3] (output) {};

    \draw [->] (input) -- node {$x(t)$} (sum);
    \draw (sum) -- (conn1);
    \draw [->] (conn1) -- (block3);
    \draw (block3) -- (conn2);
    \draw [->] (conn2) -- (block4);
    \draw [->] (conn2) -- node {$a$} (sum2);
    \draw (block4) -- (conn3);
    \draw (conn3) -- node {$b$} (conn4);
    \draw [->] (conn3) -| (sum2);
    \draw [->] (sum2) -- node[pos=0.95] {$-$} (sum);
    \draw [->] (conn3) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}
\end{center}

\section{Implementing a System in Hardware}

One of the most powerful uses of block diagrams is the implementation of a CT system in hardware. As we shall see later in the semester, designing CT systems for a particular purpose leads to a mathematical description that is equivalent to either an impulse response, or a LCCDE. We have seen how these can be represented as block diagrams. Once we have reduced a system to blocks consisting of simple operations, we can then convert the block diagram to a circuit.

\begin{tabular}{cc}

  Block & Typical Circuit\\
  \hline
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [shape=coordinate, name=signal1] at (1,0) {};
    \node [shape=coordinate, name=signal2] at (2,0) {};
    \node [output, right of=signal2] (output) {};

    \draw (input) -- node {$x(t)$} (signal1);
    \draw (signal1) -- node {$a < 0$} (signal2);
    \draw [->] (signal2) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}

  &
  \begin{circuitikz}[american voltages,scale=0.8, every node/.style={transform shape}]
    \draw
    (5,3.5) node[op amp] (opamp1) {}
    (0,4) to[R,l=$R_1$,o-] (4,4)
    (4,4) to[short] (opamp1.-)
    (opamp1.+) to[short] (3.8,2)
    (0,2) to[short,o-o] (8,2)
    (opamp1.out) to[short] (6.2,5)
    (3.5,5) to[R,l=$R_2$] (6.2,5)
    (3.5,4) to[short] (3.5,5)
    (opamp1.out) to[short,-o] (8,3.5)
    (0,4) to[open, v=$x(t)$] (0,2)
    (8,3.5) to[open, v=$y(t)$] (8,2);
  \end{circuitikz}
  \\[2em]

  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [shape=coordinate, name=signal1] at (1,0) {};
    \node [shape=coordinate, name=signal2] at (2,0) {};
    \node [output, right of=signal2] (output) {};

    \draw (input) -- node {$x(t)$} (signal1);
    \draw (signal1) -- node {$a > 1$} (signal2);
    \draw [->] (signal2) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}
  &
  \begin{circuitikz}[american voltages,scale=0.8, every node/.style={transform shape}]
    \draw
    (7,3.5) node[op amp] (opamp1) {}
    (4,0) to[short,o-o] (12,0)
    (4,4) to[short,o-] (opamp1.-)
    (opamp1.+) to[short] (5.8,1.75)
    (5.8,1.75) to[short] (8.2,1.75)
    (opamp1.out) to[R, l=$R_1$] (8.2,1.75)
    (8.2,1.75) to[R, l=$R_2$] (8.2,0)
    (opamp1.out) to[short, -o] (12,3.5)
    (4,4) to[open, v=$x(t)$] (4,0)
    (12,3.5) to[open, v=$y(t)$] (12,0);
  \end{circuitikz}
  \\[2em]

      \tikzstyle{block} = [draw, fill=gray!20, rectangle,
        minimum height=2em, minimum width=2em]
      \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
      \tikzstyle{input} = [coordinate]
      \tikzstyle{output} = [coordinate]
      \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

      \begin{tikzpicture}[auto]
        \node [input, name=input1] at (0,0) {};
        \node [input, name=input2] at (0,-1) {};
        \node [sum] at (2,0) (sum1) {$\Sigma$};
        \node [output, right of=sum1] (output) {};

        \draw [->] (input1) -- node[pos=0] {$x_1(t)$} (sum1);
        \draw [->] (input2) -| node[pos=0] {$x_2(t)$} (sum1);
        \draw [->] (sum1) -- node[pos=1] {$y(t)$} (output);
      \end{tikzpicture}
    &
      \begin{circuitikz}[american voltages,scale=0.8, every node/.style={transform shape}]
    \draw
    (9,3.5) node[op amp] (opamp1) {}
    (2,0) to[short,o-o] (12,0)
    (2,4) to[short,o-] (5,4)
    (4.5,2) to[short,o-] (5,2)
    (5,4) to[R, l=$R$] (7,4)
    (5,2) to[R, l=$R$] (7,2)
    (7,2) to[short] (7,4)
    (7,4) to[short] (opamp1.-)
    (opamp1.+) to[short] (7.8,1.75)
    (7.8,1.75) to[short] (10.2,1.75)
    (opamp1.out) to[short] (10.2,1.75)
    (opamp1.out) to[short, -o] (12,3.5)
    (2,4) to[open, v=$x_1(t)$] (2,0)
    (4.5,2) to[open, v=$x_2(t)$] (4.5,0)
    (12,3.5) to[open, v=$y(t)$] (12,0);
  \end{circuitikz}
  \\[2em]

  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node[block] at (2,0) (block1) {$-\int$};
    \node [output, right of=block1] (output) {};

    \draw [->] (input) -- node {$x(t)$} (block1);
    \draw [->] (block1) -- node[pos=1] {$y(t)$} (output);
  \end{tikzpicture}

  &
    \begin{circuitikz}[american voltages,scale=0.8, every node/.style={transform shape}]
    \draw
    (5,3.5) node[op amp] (opamp1) {}
    (0,4) to[R,l=$R$,o-] (4,4)
    (4,4) to[short] (opamp1.-)
    (opamp1.+) to[short] (3.8,2)
    (0,2) to[short,o-o] (8,2)
    (opamp1.out) to[short] (6.2,5)
    (3.5,5) to[C,l=$C$] (6.2,5)
    (3.5,4) to[short] (3.5,5)
    (opamp1.out) to[short,-o] (8,3.5)
    (0,4) to[open, v=$x(t)$] (0,2)
    (8,3.5) to[open, v=$y(t)$] (8,2);
    \end{circuitikz}\\
    \hline
\end{tabular}

\newpage
\section*{Solved Problems}

\begin{enumerate}
\item Consider a system with the following block diagram:
\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    \node [input, name=input] at (0,0) {};
    \node [block] at (4.5,0) (block1) {$\int$};
    \node [sum] at (2,0) (sum) {$\Sigma$};
    \node [output, name=feedback] at (6,0) {};
    \node [output, name=feedback2] at (6,1) {};
    \node [output, name=output] at (8,-5) {};
    \node [block] at (6,-2) (block2) {$\int$};
    \node [output, name=output2] at (6,-3) {};
    \node [block] at (6,-4) (block3) {$\int$};
    \node [output, name=output3] at (6,-5) {};
    \draw [->] (input) --  (sum);
    \draw [->] (sum) -- (block1);
    \draw (block1) -- (feedback);
    \draw (feedback) -- (feedback2);
    \draw [->] (feedback2) -| node[pos=0.95] {$-$} (sum);
    \draw [->] (output3) -- node {$b$} (output);
    \draw [->] (feedback) -- (block2);
    \draw [->] (block2) -- (block3);
    \draw [->] (output2) -| node[pos=0.95] {$-$} (sum);
    \draw (block3) -- (output3);
    \draw node at (-0.5,0) {$x(t)$};
    \draw node at (8.5,-5) {$y(t)$};
    \draw node at (4,-2.75) {$a$};
  \end{tikzpicture}
\end{center}

Determine the differential equation representation of this system.\\[1em]

\textbf{Solution:} We can convert this back to a differential equation representation as follows. First label the output of each block as a signal (called the internal states of the system), which we denote as $u(t)$, $v(t)$, $w(t)$, and $z(t)$ below.

\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
    minimum height=2em, minimum width=2em]
  \tikzstyle{sum} = [draw, fill=gray!20, circle, node distance=1cm]
  \tikzstyle{input} = [coordinate]
  \tikzstyle{output} = [coordinate]
  \tikzstyle{pinstyle} = [pin edge={to-,thin,black}]

  \begin{tikzpicture}[auto, node distance=2cm,>=latex',scale=1, every node/.style={transform shape}]
    \node [input, name=input] at (0,0) {};
    \node [block] at (4.5,0) (block1) {$\int$};
    \node [sum] at (2,0) (sum) {$\Sigma$};
    \node [output, name=feedback] at (6,0) {};
    \node [output, name=feedback2] at (6,1) {};
    \node [output, name=output] at (8,-5) {};
    \node [block] at (6,-2) (block2) {$\int$};
    \node [output, name=output2] at (6,-3) {};
    \node [block] at (6,-4) (block3) {$\int$};
    \node [output, name=output3] at (6,-5) {};
    \draw [->] (input) --  (sum);
    \draw [->] (sum) -- (block1);
    \draw (block1) -- (feedback);
    \draw (feedback) -- (feedback2);
    \draw [->] (feedback2) -| node[pos=0.95] {$-$} (sum);
    \draw [->] (output3) -- node {$b$} (output);
    \draw [->] (feedback) -- (block2);
    \draw [->] (block2) -- (block3);
    \draw [->] (output2) -| node[pos=0.95] {$-$} (sum);
    \draw (block3) -- (output3);
    \draw node at (-0.5,0) {$x(t)$};
    \draw node at (8.5,-5) {$y(t)$};
    \draw node at (4,-2.75) {$a$};
    \draw node at (6.5,0) {$u(t)$};
    \draw node at (6.5,-3) {$v(t)$};
    \draw node at (3,-0.3) {$w(t)$};
    \draw node at (5.5,-5) {$z(t)$};
  \end{tikzpicture}
\end{center}
Now we can read off the input-output relationships moving from input to output. Starting with the output of the summation
\[
w(t) = x(t) - u(t) -a\,v(t) \; .
\]
The outputs of each integrator are:
\[
u(t) = \int\limits_{-\infty}^t w(\tau) \; d\tau\;, \;
v(t) = \int\limits_{-\infty}^t u(\tau) \; d\tau\;, \mbox{ and }\;
z(t) = \int\limits_{-\infty}^t v(\tau) \; d\tau
\]
or equivalently
\[
\frac{du}{dt}(t) = w(t)\;,\; \frac{dv}{dt}(t) = u(t)\; ,\; \mbox{ and }\; \frac{dz}{dt}(t) = v(t)
\]
Finally, the output is:
\[
y(t) = b\, z(t)\; .
\]
We now do a series of derivatives and substitutions
\begin{align*}
  y(t) &= b\, z(t)\\
  \frac{dy}{dt}(t) &= b\, \frac{dz}{dt}(t)\\
  &= b\, v(t)\\
  \frac{d^2y}{dt^2}(t) &= b\, \frac{dv}{dt}(t)\\
  &= b\, u(t)\\
  \frac{d^3y}{dt^3}(t) &= b\, \frac{du}{dt}(t)\\
  &= b\, w(t)\\
  &= b\left( x(t) - u(t) -a\,v(t)\right)
\end{align*}
Rearranging the last equation to isolate the input on the right hand side gives
\[
\frac{d^3y}{dt^3}(t) + b\,u(t) +ab\,v(t) = b\,x(t)\; \mbox{ (Eqn.~1)}
\]
We can now note from above
\[
u(t) = \frac{dv}{dt}(t) =  \frac{d^2z}{dt^2}(t) = \frac{1}{b} \frac{d^2y}{dt^2}(t) \mbox{ and }
\]
\[
v(t) = \frac{dz}{dt}(t) = \frac{1}{b} \frac{dy}{dt}(t)\; .
\]
Substituting these back into Eqn.~1 gives
\[
\frac{d^3y}{dt^3}(t) + \frac{d^2y}{dt^2}(t) +a\,\frac{dy}{dt}(t) = b\,x(t)
\]
Which is a LCCDE.\\
$\blacksquare$
\end{enumerate}


\section{Feature Matching}
Feature: measured characteristic of (part of) a pattern / object\\
Goal: efficient matching

Features shoud overcome large variations in
\begin{enumerate}
	\item Viewpoint
	\item Illumination
	\item Background
	\item Occlusions
	\item Scale change
	\item Deformation
	\item Perspective Deformation
	\item Blur
\end{enumerate}

A feature should capture something \textbf{discriminative} about a well \textbf{localisable} patch of a surface

\subsection{Detector}

The detector typically yields image points. Corners are the most prominent example of so-called `Interest Points’, i.e. points that can be well localised in different views of a scene

\subsubsection{Uniqueness of a patch}
How do the patterns change upon a shift?\\
\textit{flat region}: no change in all directions\\
\textit{edge region}: no change along edge direction\\
\textit{corner} : significant change in all directions\\
\\
Compare each pixel before and after by summing up the squared differences (SSD) - this defines an error $E(u,v)$. Shift in x-direction: $u$, in y-direction:$y$.
$$ E(u,v) = \sum_{(x,y)\in W} [I(x+u, y+v) - I(x,y)]^2 $$
$$ E(u,v) = \sum_{(x,y)\in W} [u v] \underbrace{\begin{bmatrix}
I_x^2&I_xI_y\\
I_yI_x&I_y^2\\
\end{bmatrix}}_H \begin{bmatrix}
u\\
v\\
\end{bmatrix} $$
Want E(u,v) to be \textbf{large} for small shifts in \textbf{all} directions - the minimum is given by the smaller eigenvalue of H.

\subsubsection{The Harris corner detector}
$$ R = det(H) - k*trace(H)^2 $$
When $|R|$ is small, which happens when $\lambda_1$ and $\lambda_2$ are small, the region is flat.\\
When $R<0$, which happens when $\lambda_1$>>$\lambda_2$ or vice versa, the region is an edge.\\
When $|R|$ is large, which happens when $\lambda_1$ and $\lambda_2$ are large and $\lambda_1 ~\lambda_2$, the region is a corner.

\subsection{Descriptor}
Descriptors then are a vector of measurements taken around each such point

We need to describe their surrounding image patch such we can discriminate between them, i.e. we need to build a feature vector for the patch \&
Invariance under geom./phot. change

\subsubsection{Deformations under projections (geom. change)}
\begin{enumerate}
	\item similarity 4DOF
	\item affinity 6DOF
	\item projectivity 8DOF
\end{enumerate}

Complexity of the groups goes up with the generality of the viewing conditions, and so does the complexity of the group’s invariants. Fewer invariants are found going from top to bottom

\subsubsection{Photometric changes}
\begin{itemize}
	\item Contrasts (intensity differences) let the non-linear offsets cancel; hence gradients are good !
	\item Moreover the orientation of gradients in the color bands is invariant under their linear changes, as is the intensity gradient orientation in case the scale factors are identical; this is indeed relevant if the illumination changes its intensity, but not its color, which is typically assumed.
	\item But even under changing color of the illumination, in practice edge orientations tend to remain the same.
\end{itemize}

\subsection{Matching}
\begin{itemize}
	\item Interest points are matched on the basis of their descriptors
	\item E.g. nearest neighbour, based on some distance like Euclidean or Mahalanobis; good to compare against 2nd nearest neighbour: OK if difference is big; or fuzzy matching w. multiple neighbours
	\item Speed-ups by using lower-dim. descriptor space (PCA) or through some coarse-to-fine scheme (very fast schemes exist to date!)
	\item Matching of individual points typically followed by some consistency check, e.g. epipolar geometry, homograpy, or topological
\end{itemize}

\subsection{The Patch}
\subsubsection{MSER - Maximally stable extremal regions}
\begin{enumerate}
	\item Start with intensity extremum
	\item Then move intensity threshold away from its value and watch the super/sub-threshold region grow
	\item Take regions at thresholds where the growth is slowest (happens when region is bounded by strong edges)
\end{enumerate}

\subsubsection{SIFT - Scale-Invariant Feature Transform}
Is a carefully crafted interest point detector + descriptor, based on intensity gradients and invariants under similarities, not affine like so far.\\
\\
\textbf{Descriptor} is based on blob detection - at several scales - that is local extrema of the Laplacian-of-Gaussian (LoG)

\includegraphics[width=0.7\columnwidth]{pictures/sift2}

\begin{itemize}
	\item Thresholded image gradients are sampled over a grid - Dominant orientation selection
	\begin{itemize}
		\item Compute image gradients of cell
		\item Build orientation histogram
		\item Find maximum of cell $\rightarrow$ orientation of cell
	\end{itemize}
	\item Create array of orientation histograms within blocks
	\item 8 orientations x 4x4 histogram array (! in picture only 2x2) = 128 dimensions
	\item Apply weighting with a Gaussian located at the center
	\item Normalized to unit vector
\end{itemize}

\includegraphics[width=\columnwidth]{pictures/sift}

\subsubsection{SURF - efficient alternative to SIFT}
no further infos


\subsubsection{Repeated Eigenvalues}
\noindent
If $A$ has an eigenvalue with multiplicity $k$, that eigenvalue needs to generate $k$ fundamental solutions. If this eigenvalue generates $k$ linearly independent eigenvectors, then the process is much like with distinct eigenvalues. Otherwise, the matrix $A$ is defective.
\begin{theorem}
	If $\lambda$ is an eigenvalue with multiplicity $k$, and $\left\{\vec{v_1}, \ldots, \vec{v_k}\right\}$ are corresponding linearly independent eigenvectors, the the set of fundamental solutions generated by $\lambda$ is $\left\{e^{\lambda t}\vec{v_1}, \ldots, e^{\lambda t}\vec{v_k}\right\}$.
\end{theorem}

\ifodd\includeLinearSystemsExamples\input{./linearSystems/homogeneousSystems/repeatedEigenvalues_example.tex}\fi