Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 282 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.8168. Min Range: -3.8168

Max Act: 3.8168. Min Act: -0.0000

Data Index: 1074055 (C4 (Web Text))

Max Activating Token Index: 18

Text #1

Max Range: 3.8168. Min Range: -3.8168

Max Act: 3.2396. Min Act: -0.0000

Data Index: 232463 (C4 (Web Text))

Max Activating Token Index: 642

Text #2

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.7051. Min Act: -0.0000

Data Index: 132902 (C4 (Web Text))

Max Activating Token Index: 455

Text #3

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.4824. Min Act: -0.0000

Data Index: 8239 (C4 (Web Text))

Max Activating Token Index: 839

Text #4

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.5322. Min Act: -0.0000

Data Index: 237735 (C4 (Web Text))

Max Activating Token Index: 836

Text #5

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.4018. Min Act: -0.0000

Data Index: 410019 (C4 (Web Text))

Max Activating Token Index: 227

Text #6

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.6320. Min Act: -0.0000

Data Index: 1330529 (C4 (Web Text))

Max Activating Token Index: 83

Text #7

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.2336. Min Act: -0.0000

Data Index: 535750 (C4 (Web Text))

Max Activating Token Index: 602

Text #8

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.1432. Min Act: -0.0000

Data Index: 776862 (C4 (Web Text))

Max Activating Token Index: 200

Text #9

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.3909. Min Act: -0.0000

Data Index: 925112 (C4 (Web Text))

Max Activating Token Index: 904

Text #10

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.1360. Min Act: -0.0000

Data Index: 1140897 (C4 (Web Text))

Max Activating Token Index: 934

Text #11

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.3478. Min Act: -0.0000

Data Index: 26459 (C4 (Web Text))

Max Activating Token Index: 961

Text #12

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.0991. Min Act: -0.0000

Data Index: 417515 (C4 (Web Text))

Max Activating Token Index: 33

Text #13

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.2109. Min Act: -0.0000

Data Index: 1310145 (C4 (Web Text))

Max Activating Token Index: 347

Text #14

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.3335. Min Act: -0.0000

Data Index: 355595 (C4 (Web Text))

Max Activating Token Index: 165

Text #15

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.2863. Min Act: -0.0000

Data Index: 244956 (C4 (Web Text))

Max Activating Token Index: 774

Text #16

Max Range: 3.8168. Min Range: -3.8168

Max Act: 1.9307. Min Act: -0.0000

Data Index: 849267 (C4 (Web Text))

Max Activating Token Index: 254

Text #17

Max Range: 3.8168. Min Range: -3.8168

Max Act: 1.9005. Min Act: -0.0000

Data Index: 1224230 (C4 (Web Text))

Max Activating Token Index: 304

Text #18

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.0582. Min Act: -0.0000

Data Index: 649781 (C4 (Web Text))

Max Activating Token Index: 280

Text #19

Max Range: 3.8168. Min Range: -3.8168

Max Act: 2.0310. Min Act: -0.0000

Data Index: 490443 (C4 (Web Text))

Max Activating Token Index: 370