Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 750 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1520. Min Range: -2.1520

Max Act: 2.1520. Min Act: -0.0000

Data Index: 1244301 (C4 (Web Text))

Max Activating Token Index: 813

Text #1

Max Range: 2.1520. Min Range: -2.1520

Max Act: 2.0246. Min Act: -0.0000

Data Index: 1206147 (C4 (Web Text))

Max Activating Token Index: 853

Text #2

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.8512. Min Act: -0.0000

Data Index: 36683 (C4 (Web Text))

Max Activating Token Index: 223

Text #3

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.8181. Min Act: -0.0000

Data Index: 1262960 (C4 (Web Text))

Max Activating Token Index: 850

Text #4

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.7521. Min Act: -0.0000

Data Index: 305413 (C4 (Web Text))

Max Activating Token Index: 396

Text #5

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.8460. Min Act: -0.0000

Data Index: 1032666 (C4 (Web Text))

Max Activating Token Index: 388

Text #6

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.7576. Min Act: -0.0000

Data Index: 1300618 (C4 (Web Text))

Max Activating Token Index: 288

Text #7

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.6373. Min Act: -0.0000

Data Index: 1167980 (C4 (Web Text))

Max Activating Token Index: 469

Text #8

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.6498. Min Act: -0.0000

Data Index: 505869 (C4 (Web Text))

Max Activating Token Index: 390

Text #9

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.6457. Min Act: -0.0000

Data Index: 1342824 (C4 (Web Text))

Max Activating Token Index: 412

Text #10

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.6897. Min Act: -0.0000

Data Index: 7071 (C4 (Web Text))

Max Activating Token Index: 701

Text #11

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.4712. Min Act: -0.0000

Data Index: 274783 (C4 (Web Text))

Max Activating Token Index: 833

Text #12

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.4003. Min Act: -0.0000

Data Index: 1156241 (C4 (Web Text))

Max Activating Token Index: 983

Text #13

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.5821. Min Act: -0.0000

Data Index: 1017774 (C4 (Web Text))

Max Activating Token Index: 783

Text #14

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.5015. Min Act: -0.0000

Data Index: 224492 (C4 (Web Text))

Max Activating Token Index: 454

Text #15

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.4166. Min Act: -0.0000

Data Index: 410409 (C4 (Web Text))

Max Activating Token Index: 529

Text #16

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.4121. Min Act: -0.0000

Data Index: 292278 (C4 (Web Text))

Max Activating Token Index: 361

Text #17

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.3083. Min Act: -0.0000

Data Index: 959831 (C4 (Web Text))

Max Activating Token Index: 821

Text #18

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.3387. Min Act: -0.0000

Data Index: 481891 (C4 (Web Text))

Max Activating Token Index: 501

Text #19

Max Range: 2.1520. Min Range: -2.1520

Max Act: 1.4227. Min Act: -0.0000

Data Index: 120652 (C4 (Web Text))

Max Activating Token Index: 486