TFGENZOO.optimizers.transformer_schedule module

class TFGENZOO.optimizers.transformer_schedule.CustomSchedule(d_model, warmup_steps=4000)

Bases: tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule

for warmup learning rate ref: https://www.tensorflow.org/tutorials/text/transformer