Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4664 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.4355. Min Act: -0.0000

Data Index: 681684 (C4 (Web Text))

Max Activating Token Index: 854

Text #1

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.5401. Min Act: -0.0000

Data Index: 364679 (C4 (Web Text))

Max Activating Token Index: 95

Text #2

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.3401. Min Act: -0.0000

Data Index: 127273 (C4 (Web Text))

Max Activating Token Index: 80

Text #3

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.3213. Min Act: -0.0000

Data Index: 298332 (C4 (Web Text))

Max Activating Token Index: 196

Text #4

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.4701. Min Act: -0.0000

Data Index: 216109 (C4 (Web Text))

Max Activating Token Index: 889

Text #5

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.0923. Min Act: -0.0000

Data Index: 335247 (C4 (Web Text))

Max Activating Token Index: 854

Text #6

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9892. Min Act: -0.0000

Data Index: 759505 (C4 (Web Text))

Max Activating Token Index: 640

Text #7

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9744. Min Act: -0.0000

Data Index: 1311471 (C4 (Web Text))

Max Activating Token Index: 125

Text #8

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9346. Min Act: -0.0000

Data Index: 1171431 (C4 (Web Text))

Max Activating Token Index: 233

Text #9

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9120. Min Act: -0.0000

Data Index: 1013607 (C4 (Web Text))

Max Activating Token Index: 70

Text #10

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.0401. Min Act: -0.0000

Data Index: 289425 (C4 (Web Text))

Max Activating Token Index: 339

Text #11

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.0554. Min Act: -0.0000

Data Index: 1321231 (C4 (Web Text))

Max Activating Token Index: 353

Text #12

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.1763. Min Act: -0.0000

Data Index: 627686 (C4 (Web Text))

Max Activating Token Index: 84

Text #13

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.1779. Min Act: -0.0000

Data Index: 1123946 (C4 (Web Text))

Max Activating Token Index: 625

Text #14

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9807. Min Act: -0.0000

Data Index: 588767 (C4 (Web Text))

Max Activating Token Index: 705

Text #15

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.9940. Min Act: -0.0000

Data Index: 666736 (C4 (Web Text))

Max Activating Token Index: 620

Text #16

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.7900. Min Act: -0.0000

Data Index: 1349774 (C4 (Web Text))

Max Activating Token Index: 149

Text #17

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.1515. Min Act: -0.0000

Data Index: 68757 (C4 (Web Text))

Max Activating Token Index: 188

Text #18

Max Range: 4.5401. Min Range: -4.5401

Max Act: 3.8242. Min Act: -0.0000

Data Index: 730096 (C4 (Web Text))

Max Activating Token Index: 151

Text #19

Max Range: 4.5401. Min Range: -4.5401

Max Act: 4.1385. Min Act: -0.0000

Data Index: 903861 (C4 (Web Text))

Max Activating Token Index: 823