Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4152 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4750. Min Act: 0.0001

Data Index: 459370 (C4 (Web Text))

Max Activating Token Index: 928

Text #1

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4662. Min Act: 0.0000

Data Index: 612321 (C4 (Web Text))

Max Activating Token Index: 195

Text #2

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4588. Min Act: -0.0000

Data Index: 1317703 (C4 (Web Text))

Max Activating Token Index: 62

Text #3

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4283. Min Act: -0.0000

Data Index: 536360 (C4 (Web Text))

Max Activating Token Index: 375

Text #4

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4180. Min Act: -0.0000

Data Index: 266053 (C4 (Web Text))

Max Activating Token Index: 979

Text #5

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4377. Min Act: 0.0001

Data Index: 837129 (C4 (Web Text))

Max Activating Token Index: 254

Text #6

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4158. Min Act: -0.0000

Data Index: 379558 (C4 (Web Text))

Max Activating Token Index: 506

Text #7

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4169. Min Act: -0.0000

Data Index: 298453 (C4 (Web Text))

Max Activating Token Index: 574

Text #8

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3981. Min Act: 0.0000

Data Index: 274501 (C4 (Web Text))

Max Activating Token Index: 825

Text #9

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4067. Min Act: 0.0000

Data Index: 543239 (C4 (Web Text))

Max Activating Token Index: 390

Text #10

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4045. Min Act: 0.0000

Data Index: 920173 (C4 (Web Text))

Max Activating Token Index: 376

Text #11

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.4022. Min Act: 0.0001

Data Index: 1254835 (C4 (Web Text))

Max Activating Token Index: 874

Text #12

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3949. Min Act: -0.0000

Data Index: 964376 (C4 (Web Text))

Max Activating Token Index: 475

Text #13

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3912. Min Act: -0.0000

Data Index: 784157 (C4 (Web Text))

Max Activating Token Index: 609

Text #14

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3890. Min Act: -0.0000

Data Index: 568891 (C4 (Web Text))

Max Activating Token Index: 257

Text #15

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3669. Min Act: -0.0000

Data Index: 943855 (C4 (Web Text))

Max Activating Token Index: 197

Text #16

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3648. Min Act: 0.0000

Data Index: 741118 (C4 (Web Text))

Max Activating Token Index: 690

Text #17

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3373. Min Act: -0.0000

Data Index: 858059 (C4 (Web Text))

Max Activating Token Index: 740

Text #18

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3588. Min Act: -0.0000

Data Index: 386646 (C4 (Web Text))

Max Activating Token Index: 501

Text #19

Max Range: 0.4750. Min Range: -0.4750

Max Act: 0.3613. Min Act: 0.0000

Data Index: 503257 (C4 (Web Text))

Max Activating Token Index: 435