Currently, Fermigrad supports activations like ReLU and tanh, but the original not published picograd I was working on had more activation functions, like sigmoid. It should be added back to support more neural network use cases.
Proposed Implementation:
def sigmoid(self):
s = 1 / (1 + math.exp(-self.data))
out = Value(s, (self,), 'sigmoid')
def _backward():
self.grad += s * (1 - s) * out.grad
out._backward = _backward
return out
Tasks:
Currently, Fermigrad supports activations like ReLU and tanh, but the original not published picograd I was working on had more activation functions, like sigmoid. It should be added back to support more neural network use cases.
Proposed Implementation:
Tasks: