Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1930 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6988. Min Act: -0.0003

Data Index: 1295002 (C4 (Web Text))

Max Activating Token Index: 60

Text #1

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.7461. Min Act: -0.0002

Data Index: 643176 (C4 (Web Text))

Max Activating Token Index: 541

Text #2

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.7227. Min Act: -0.0003

Data Index: 376770 (C4 (Web Text))

Max Activating Token Index: 193

Text #3

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6991. Min Act: -0.0002

Data Index: 1252607 (C4 (Web Text))

Max Activating Token Index: 167

Text #4

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6940. Min Act: -0.0002

Data Index: 683108 (C4 (Web Text))

Max Activating Token Index: 472

Text #5

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6568. Min Act: -0.0002

Data Index: 579300 (C4 (Web Text))

Max Activating Token Index: 367

Text #6

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6239. Min Act: -0.0003

Data Index: 537511 (C4 (Web Text))

Max Activating Token Index: 489

Text #7

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.5975. Min Act: -0.0003

Data Index: 435070 (C4 (Web Text))

Max Activating Token Index: 81

Text #8

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.6216. Min Act: -0.0003

Data Index: 679124 (C4 (Web Text))

Max Activating Token Index: 264

Text #9

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.5240. Min Act: -0.0002

Data Index: 982956 (C4 (Web Text))

Max Activating Token Index: 599

Text #10

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.5197. Min Act: -0.0002

Data Index: 826782 (C4 (Web Text))

Max Activating Token Index: 491

Text #11

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4465. Min Act: -0.0002

Data Index: 1302765 (C4 (Web Text))

Max Activating Token Index: 57

Text #12

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.5059. Min Act: -0.0002

Data Index: 231387 (C4 (Web Text))

Max Activating Token Index: 541

Text #13

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4855. Min Act: -0.0003

Data Index: 24976 (C4 (Web Text))

Max Activating Token Index: 384

Text #14

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4509. Min Act: -0.0002

Data Index: 280935 (C4 (Web Text))

Max Activating Token Index: 675

Text #15

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4458. Min Act: -0.0003

Data Index: 1343759 (C4 (Web Text))

Max Activating Token Index: 739

Text #16

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4229. Min Act: -0.0002

Data Index: 187527 (C4 (Web Text))

Max Activating Token Index: 672

Text #17

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4042. Min Act: -0.0002

Data Index: 748719 (C4 (Web Text))

Max Activating Token Index: 679

Text #18

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.4061. Min Act: -0.0002

Data Index: 1294769 (C4 (Web Text))

Max Activating Token Index: 782

Text #19

Max Range: 1.7461. Min Range: -1.7461

Max Act: 1.3569. Min Act: -0.0002

Data Index: 262398 (C4 (Web Text))

Max Activating Token Index: 267