set bias=True for linear layer (#527)

Summary:
Pull Request resolved: #527

Pull Request resolved: #526

1. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's  output module and LLAVA, they are using bias=True (which is default value in Linear).
2. add configuration of using MLP instead of attention pooler for vision adapter;

Reviewed By: Bellaktris

Differential Revision:
D55897450

Privacy Context Container: 303860477774201

fbshipit-source-id: 8e012b0c3d37566364f216dbfa8aec389142afe1

Author: zhangtemplar

tests/test_utils.py

@@ -192,8 +192,18 @@ def assert_expected_namedtuple(
 
 
 def init_weights_with_constant(model: nn.Module, constant: float = 1.0) -> None:
-    for p in model.parameters():
+    for n, p in model.named_parameters():
         nn.init.constant_(p, constant)
+        # reduce the change to the tests
+        for k in {
+            "text_projection.bias",
+            "pooled_projection.bias",
+            "output_projection.bias",
+            "vision_proj.bias",
+        }:
+            if n.endswith(k):
+                nn.init.constant_(p, 0.0)
+                break
 
 
 def tensor_hash(x: torch.tensor, scaling=0.05, buckets=1000) -> torch.tensor: