Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 326 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.2456. Min Range: -2.2456

Max Act: 2.2456. Min Act: -0.0000

Data Index: 338903 (C4 (Web Text))

Max Activating Token Index: 189

Text #1

Max Range: 2.2456. Min Range: -2.2456

Max Act: 2.0744. Min Act: -0.0000

Data Index: 278830 (C4 (Web Text))

Max Activating Token Index: 291

Text #2

Max Range: 2.2456. Min Range: -2.2456

Max Act: 2.0385. Min Act: -0.0000

Data Index: 1350699 (C4 (Web Text))

Max Activating Token Index: 701

Text #3

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.8719. Min Act: -0.0000

Data Index: 565842 (C4 (Web Text))

Max Activating Token Index: 204

Text #4

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.6304. Min Act: -0.0000

Data Index: 1236634 (C4 (Web Text))

Max Activating Token Index: 192

Text #5

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.8227. Min Act: -0.0000

Data Index: 316871 (C4 (Web Text))

Max Activating Token Index: 161

Text #6

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5566. Min Act: -0.0000

Data Index: 1144469 (C4 (Web Text))

Max Activating Token Index: 74

Text #7

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.8091. Min Act: -0.0000

Data Index: 461091 (C4 (Web Text))

Max Activating Token Index: 886

Text #8

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.6320. Min Act: -0.0000

Data Index: 155305 (C4 (Web Text))

Max Activating Token Index: 1000

Text #9

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5806. Min Act: -0.0000

Data Index: 824985 (C4 (Web Text))

Max Activating Token Index: 408

Text #10

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.7298. Min Act: -0.0000

Data Index: 591100 (C4 (Web Text))

Max Activating Token Index: 237

Text #11

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.6310. Min Act: -0.0000

Data Index: 402071 (C4 (Web Text))

Max Activating Token Index: 157

Text #12

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.7232. Min Act: -0.0000

Data Index: 1266681 (C4 (Web Text))

Max Activating Token Index: 339

Text #13

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.6298. Min Act: -0.0000

Data Index: 43314 (C4 (Web Text))

Max Activating Token Index: 290

Text #14

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5570. Min Act: -0.0000

Data Index: 312493 (C4 (Web Text))

Max Activating Token Index: 47

Text #15

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.4823. Min Act: -0.0000

Data Index: 494240 (C4 (Web Text))

Max Activating Token Index: 313

Text #16

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5735. Min Act: -0.0000

Data Index: 1137161 (C4 (Web Text))

Max Activating Token Index: 967

Text #17

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5197. Min Act: -0.0000

Data Index: 1208986 (C4 (Web Text))

Max Activating Token Index: 962

Text #18

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.4412. Min Act: -0.0000

Data Index: 1069121 (C4 (Web Text))

Max Activating Token Index: 287

Text #19

Max Range: 2.2456. Min Range: -2.2456

Max Act: 1.5050. Min Act: -0.0000

Data Index: 710927 (C4 (Web Text))

Max Activating Token Index: 156