Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 131 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0229. Min Act: -0.0005

Data Index: 158646 (C4 (Web Text))

Max Activating Token Index: 353

Text #1

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0230. Min Act: -0.0005

Data Index: 57412 (C4 (Web Text))

Max Activating Token Index: 108

Text #2

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0230. Min Act: -0.0005

Data Index: 104865 (C4 (Web Text))

Max Activating Token Index: 740

Text #3

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0239. Min Act: -0.0005

Data Index: 600136 (C4 (Web Text))

Max Activating Token Index: 680

Text #4

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0235. Min Act: -0.0005

Data Index: 608966 (C4 (Web Text))

Max Activating Token Index: 249

Text #5

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0233. Min Act: -0.0005

Data Index: 805447 (C4 (Web Text))

Max Activating Token Index: 977

Text #6

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0225. Min Act: -0.0005

Data Index: 155057 (C4 (Web Text))

Max Activating Token Index: 415

Text #7

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0230. Min Act: -0.0005

Data Index: 325886 (C4 (Web Text))

Max Activating Token Index: 918

Text #8

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0230. Min Act: -0.0005

Data Index: 922203 (C4 (Web Text))

Max Activating Token Index: 685

Text #9

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0235. Min Act: -0.0005

Data Index: 613724 (C4 (Web Text))

Max Activating Token Index: 346

Text #10

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0221. Min Act: -0.0005

Data Index: 1013298 (C4 (Web Text))

Max Activating Token Index: 984

Text #11

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0211. Min Act: -0.0005

Data Index: 1155060 (C4 (Web Text))

Max Activating Token Index: 934

Text #12

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0219. Min Act: -0.0005

Data Index: 453223 (C4 (Web Text))

Max Activating Token Index: 996

Text #13

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0228. Min Act: -0.0005

Data Index: 742556 (C4 (Web Text))

Max Activating Token Index: 103

Text #14

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0210. Min Act: -0.0005

Data Index: 910671 (C4 (Web Text))

Max Activating Token Index: 374

Text #15

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0224. Min Act: -0.0005

Data Index: 834978 (C4 (Web Text))

Max Activating Token Index: 481

Text #16

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0205. Min Act: -0.0005

Data Index: 1179907 (C4 (Web Text))

Max Activating Token Index: 488

Text #17

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0215. Min Act: -0.0005

Data Index: 261702 (C4 (Web Text))

Max Activating Token Index: 232

Text #18

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0208. Min Act: -0.0005

Data Index: 945943 (C4 (Web Text))

Max Activating Token Index: 475

Text #19

Max Range: 0.0239. Min Range: -0.0239

Max Act: 0.0218. Min Act: -0.0005

Data Index: 1122720 (C4 (Web Text))

Max Activating Token Index: 838