Bagaimana cara membuat Keras Dense Layer dengan tensor 3D sebagai input untuk Softmax Fully Connected Layer ini?

Question

Saya sedang mengerjakan masalah khusus, dan saya harus mengubah lapisan yang terhubung sepenuhnya (Padat dengan softmax), kode model saya adalah seperti ini (dengan Kerangka Keras):

.......
batch_size = 8
inputs = tf.random.uniform(shape=[batch_size,1024,256],dtype=tf.dtypes.float32)
preds = Dense(num_classes,activation='softmax')(x) #final layer with softmax activation
....
model = Model(inputs=base_model.input,outputs=preds)

Jadi, saya harus mengubah Code of Dense Layer untuk mengeluarkan Tensor probabilitas dengan bentuk [batch_size, 1024, num_classes], tanpa menggunakan for loop, saya membutuhkannya untuk dioptimalkan dan bukan fungsi waktu yang memakan waktu

Versi kode Dense yang ingin saya ubah:

class Dense(Layer):
"""Just your regular densely-connected NN layer.

`Dense` implements the operation:
`output = activation(dot(input, kernel) + bias)`
where `activation` is the element-wise activation function
passed as the `activation` argument, `kernel` is a weights matrix
created by the layer, and `bias` is a bias vector created by the layer
(only applicable if `use_bias` is `True`).

Note: if the input to the layer has a rank greater than 2, then
it is flattened prior to the initial dot product with `kernel`.

# Example

```python
    # as first layer in a sequential model:
    model = Sequential()
    model.add(Dense(32, input_shape=(16,)))
    # now the model will take as input arrays of shape (*, 16)
    # and output arrays of shape (*, 32)

    # after the first layer, you don't need to specify
    # the size of the input anymore:
    model.add(Dense(32))
```

# Arguments
    units: Positive integer, dimensionality of the output space.
    activation: Activation function to use
        (see [activations](../activations.md)).
        If you don't specify anything, no activation is applied
        (ie. "linear" activation: `a(x) = x`).
    use_bias: Boolean, whether the layer uses a bias vector.
    kernel_initializer: Initializer for the `kernel` weights matrix
        (see [initializers](../initializers.md)).
    bias_initializer: Initializer for the bias vector
        (see [initializers](../initializers.md)).
    kernel_regularizer: Regularizer function applied to
        the `kernel` weights matrix
        (see [regularizer](../regularizers.md)).
    bias_regularizer: Regularizer function applied to the bias vector
        (see [regularizer](../regularizers.md)).
    activity_regularizer: Regularizer function applied to
        the output of the layer (its "activation").
        (see [regularizer](../regularizers.md)).
    kernel_constraint: Constraint function applied to
        the `kernel` weights matrix
        (see [constraints](../constraints.md)).
    bias_constraint: Constraint function applied to the bias vector
        (see [constraints](../constraints.md)).

# Input shape
    nD tensor with shape: `(batch_size, ..., input_dim)`.
    The most common situation would be
    a 2D input with shape `(batch_size, input_dim)`.

# Output shape
    nD tensor with shape: `(batch_size, ..., units)`.
    For instance, for a 2D input with shape `(batch_size, input_dim)`,
    the output would have shape `(batch_size, units)`.
"""

def __init__(self, units,
             activation=None,
             use_bias=True,
             kernel_initializer='glorot_uniform',
             bias_initializer='zeros',
             kernel_regularizer=None,
             bias_regularizer=None,
             activity_regularizer=None,
             kernel_constraint=None,
             bias_constraint=None,
             **kwargs):
    if 'input_shape' not in kwargs and 'input_dim' in kwargs:
        kwargs['input_shape'] = (kwargs.pop('input_dim'),)
    super(Dense, self).__init__(**kwargs)
    self.units = units
    self.activation = activations.get(activation)
    self.use_bias = use_bias
    self.kernel_initializer = initializers.get(kernel_initializer)
    self.bias_initializer = initializers.get(bias_initializer)
    self.kernel_regularizer = regularizers.get(kernel_regularizer)
    self.bias_regularizer = regularizers.get(bias_regularizer)
    self.activity_regularizer = regularizers.get(activity_regularizer)
    self.kernel_constraint = constraints.get(kernel_constraint)
    self.bias_constraint = constraints.get(bias_constraint)
    self.input_spec = InputSpec(min_ndim=2)
    self.supports_masking = True

def build(self, input_shape):
    assert len(input_shape) >= 2 
    input_dim = input_shape[-1]  

    self.kernel = self.add_weight(shape=(input_dim, self.units),
                                  initializer=self.kernel_initializer,
                                  name='kernel',
                                  regularizer=self.kernel_regularizer,
                                  constraint=self.kernel_constraint)
    if self.use_bias:
        self.bias = self.add_weight(shape=(self.units,),
                                    initializer=self.bias_initializer,
                                    name='bias',
                                    regularizer=self.bias_regularizer,
                                    constraint=self.bias_constraint)
    else:
        self.bias = None
    self.input_spec = InputSpec(min_ndim=2, axes={-1: input_dim})
    self.built = True

def call(self, inputs):
    output = K.dot(inputs, self.kernel)
    if self.use_bias:
        output = K.bias_add(output, self.bias)
    if self.activation is not None:
        output = self.activation(output)
    return output

def compute_output_shape(self, input_shape):
    assert input_shape and len(input_shape) >= 2
    assert input_shape[-1]
    output_shape = list(input_shape)
    output_shape[-1] = self.units
    return tuple(output_shape)

def get_config(self):
    config = {
        'units': self.units,
        'activation': activations.serialize(self.activation),
        'use_bias': self.use_bias,
        'kernel_initializer': initializers.serialize(self.kernel_initializer),
        'bias_initializer': initializers.serialize(self.bias_initializer),
        'kernel_regularizer': regularizers.serialize(self.kernel_regularizer),
        'bias_regularizer': regularizers.serialize(self.bias_regularizer),
        'activity_regularizer': regularizers.serialize(self.activity_regularizer),
        'kernel_constraint': constraints.serialize(self.kernel_constraint),
        'bias_constraint': constraints.serialize(self.bias_constraint)
    }
    base_config = super(Dense, self).get_config()
    return dict(list(base_config.items()) + list(config.items()))

Answer 1

Ada tiga cara berbeda untuk melakukan ini (yang dapat saya pikirkan). Jika Anda ingin memiliki satu lapisan padat, yang memetakan vektor 256 elemen ke vektor num_classeselemen, dan menerapkannya di seluruh kumpulan data Anda (yaitu, menggunakan 256 x num_classesmatriks bobot yang sama untuk setiap sampel), maka Anda tidak tidak perlu melakukan sesuatu yang khusus, cukup gunakan Denselayer biasa :

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.layers import Dense

batch_size = 8
num_classes = 10
inp = Input(shape=(1024, 256))
layer = Dense(num_classes, activation='softmax')
out = layer(inp)
print(out.shape)
# (None, 1024, 10)
print(layer.count_params())
# 2570

Cara lain adalah memiliki satu Denselapisan besar yang mengambil semua 1024 * 256nilai pada saat yang sama dan menghasilkan semua 1024 * num_classesnilai pada keluaran, yaitu lapisan dengan matriks bobot dengan bentuk (1024 * 256) x (1024 * num_classes)(dalam urutan jika memori gigabyte!). Ini juga mudah dilakukan, meskipun tampaknya tidak seperti yang Anda butuhkan:

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.layers import Flatten, Dense, Reshape, Softmax

batch_size = 8
num_classes = 10
inp = Input(shape=(1024, 256))
res = Flatten()(inp)
# This takes _a lot_ of memory!
layer = Dense(1024 * num_classes, activation=None)
out_res = layer(res)
# Apply softmax after reshaping
out_preact = Reshape((-1, num_classes))(out_res)
out = Softmax()(out_preact)
print(out.shape)
# (None, 1024, 10)
print(layer.count_params())
# 2684364800

Terakhir, Anda mungkin ingin memiliki satu set 1024 matriks bobot, masing-masing diterapkan ke sampel terkait dalam masukan, yang akan menyiratkan larik bobot dengan bentuk (1024, 256, num_classes). Saya rasa ini tidak dapat dilakukan dengan salah satu lapisan Keras standar (atau tidak tahu cara melakukannya) ¹ , tetapi cukup mudah untuk menulis lapisan khusus berdasarkan Denseuntuk melakukannya:

import tensorflow as tf
from tensorflow.keras.layers import Dense, InputSpec

class Dense2D(Dense):
    def __init__(self, *args, **kwargs):
        super(Dense2D, self).__init__(*args, **kwargs)

    def build(self, input_shape):
        assert len(input_shape) >= 3
        input_dim1 = input_shape[-2]
        input_dim2 = input_shape[-1]

        self.kernel = self.add_weight(shape=(input_dim1, input_dim2, self.units),
                                      initializer=self.kernel_initializer,
                                      name='kernel',
                                      regularizer=self.kernel_regularizer,
                                      constraint=self.kernel_constraint)
        if self.use_bias:
            self.bias = self.add_weight(shape=(input_dim1, self.units),
                                        initializer=self.bias_initializer,
                                        name='bias',
                                        regularizer=self.bias_regularizer,
                                        constraint=self.bias_constraint)
        else:
            self.bias = None
        self.input_spec = InputSpec(min_ndim=3, axes={-2: input_dim1, -1: input_dim2})
        self.built = True

    def call(self, inputs):
        # Multiply each set of weights with each input element
        output = tf.einsum('...ij,ijk->...ik', inputs, self.kernel)
        if self.use_bias:
            output += self.bias
        if self.activation is not None:
            output = self.activation(output)
        return output

    def compute_output_shape(self, input_shape):
        assert input_shape and len(input_shape) >= 3
        assert input_shape[-1]
        output_shape = list(input_shape)
        output_shape[-1] = self.units
        return tuple(output_shape)

Anda kemudian akan menggunakannya seperti ini:

import tensorflow as tf
from tensorflow.keras import Input

batch_size = 8
num_classes = 10
inp = Input(shape=(1024, 256))
layer = Dense2D(num_classes, activation='softmax')
out = layer(inp)
print(out.shape)
# (None, 1024, 10)
print(layer.count_params())
# 2631680

---

¹ : Seperti yang hari ini tunjukkan di komentar, Anda sebenarnya dapat menggunakan LocallyConnected1Dlapisan untuk melakukan hal yang sama seperti yang saya coba lakukan dengan Dense2Dlapisan saya . Ini sesederhana ini:

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.layers import LocallyConnected1D

batch_size = 8
num_classes = 10
inp = Input(shape=(1024, 256))
layer = LocallyConnected1D(num_classes, 1, activation='softmax')
out = layer(inp)
print(out.shape)
# (None, 1024, 10)
print(layer.count_params())
# 2631680