Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1059 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.6250. Min Act: -0.0000

Data Index: 135200 (C4 (Web Text))

Max Activating Token Index: 149

Text #1

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.3459. Min Act: -0.0000

Data Index: 74891 (C4 (Web Text))

Max Activating Token Index: 701

Text #2

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.3858. Min Act: -0.0000

Data Index: 701834 (C4 (Web Text))

Max Activating Token Index: 859

Text #3

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1881. Min Act: -0.0000

Data Index: 28998 (C4 (Web Text))

Max Activating Token Index: 520

Text #4

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1963. Min Act: -0.0000

Data Index: 841895 (C4 (Web Text))

Max Activating Token Index: 12

Text #5

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.2764. Min Act: -0.0000

Data Index: 297003 (C4 (Web Text))

Max Activating Token Index: 217

Text #6

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.9904. Min Act: -0.0000

Data Index: 685912 (C4 (Web Text))

Max Activating Token Index: 90

Text #7

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0881. Min Act: -0.0000

Data Index: 727757 (C4 (Web Text))

Max Activating Token Index: 705

Text #8

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0576. Min Act: -0.0000

Data Index: 803678 (C4 (Web Text))

Max Activating Token Index: 737

Text #9

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1020. Min Act: -0.0000

Data Index: 902669 (C4 (Web Text))

Max Activating Token Index: 256

Text #10

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0304. Min Act: -0.0000

Data Index: 1039683 (C4 (Web Text))

Max Activating Token Index: 472

Text #11

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1125. Min Act: -0.0000

Data Index: 922649 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.9491. Min Act: -0.0000

Data Index: 244469 (C4 (Web Text))

Max Activating Token Index: 117

Text #13

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1201. Min Act: -0.0000

Data Index: 951117 (C4 (Web Text))

Max Activating Token Index: 710

Text #14

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.8543. Min Act: -0.0000

Data Index: 1179050 (C4 (Web Text))

Max Activating Token Index: 635

Text #15

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0418. Min Act: -0.0000

Data Index: 274591 (C4 (Web Text))

Max Activating Token Index: 450

Text #16

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.0289. Min Act: -0.0000

Data Index: 559580 (C4 (Web Text))

Max Activating Token Index: 869

Text #17

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.8398. Min Act: -0.0000

Data Index: 931788 (C4 (Web Text))

Max Activating Token Index: 645

Text #18

Max Range: 2.6250. Min Range: -2.6250

Max Act: 1.9266. Min Act: -0.0000

Data Index: 734160 (C4 (Web Text))

Max Activating Token Index: 49

Text #19

Max Range: 2.6250. Min Range: -2.6250

Max Act: 2.1409. Min Act: -0.0000

Data Index: 979335 (C4 (Web Text))

Max Activating Token Index: 292