Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4991 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1827. Min Range: -1.1827

Max Act: 1.1827. Min Act: -0.0000

Data Index: 1091546 (C4 (Web Text))

Max Activating Token Index: 534

Text #1

Max Range: 1.1827. Min Range: -1.1827

Max Act: 1.0621. Min Act: -0.0000

Data Index: 1016414 (C4 (Web Text))

Max Activating Token Index: 964

Text #2

Max Range: 1.1827. Min Range: -1.1827

Max Act: 1.0094. Min Act: -0.0000

Data Index: 868207 (C4 (Web Text))

Max Activating Token Index: 149

Text #3

Max Range: 1.1827. Min Range: -1.1827

Max Act: 1.0256. Min Act: -0.0000

Data Index: 1153012 (C4 (Web Text))

Max Activating Token Index: 742

Text #4

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.9846. Min Act: -0.0000

Data Index: 551142 (C4 (Web Text))

Max Activating Token Index: 108

Text #5

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.8207. Min Act: -0.0000

Data Index: 1339039 (C4 (Web Text))

Max Activating Token Index: 146

Text #6

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7792. Min Act: -0.0000

Data Index: 368940 (C4 (Web Text))

Max Activating Token Index: 506

Text #7

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7186. Min Act: -0.0000

Data Index: 892376 (C4 (Web Text))

Max Activating Token Index: 288

Text #8

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.8040. Min Act: -0.0000

Data Index: 369010 (C4 (Web Text))

Max Activating Token Index: 731

Text #9

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7474. Min Act: -0.0000

Data Index: 893527 (C4 (Web Text))

Max Activating Token Index: 335

Text #10

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7142. Min Act: -0.0000

Data Index: 288872 (C4 (Web Text))

Max Activating Token Index: 86

Text #11

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7071. Min Act: -0.0000

Data Index: 662761 (C4 (Web Text))

Max Activating Token Index: 813

Text #12

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7098. Min Act: -0.0000

Data Index: 717852 (C4 (Web Text))

Max Activating Token Index: 154

Text #13

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7091. Min Act: -0.0000

Data Index: 809393 (C4 (Web Text))

Max Activating Token Index: 582

Text #14

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7026. Min Act: -0.0000

Data Index: 1019665 (C4 (Web Text))

Max Activating Token Index: 602

Text #15

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.6956. Min Act: -0.0000

Data Index: 882050 (C4 (Web Text))

Max Activating Token Index: 1006

Text #16

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.6788. Min Act: -0.0000

Data Index: 88404 (C4 (Web Text))

Max Activating Token Index: 290

Text #17

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.6711. Min Act: -0.0000

Data Index: 222477 (C4 (Web Text))

Max Activating Token Index: 947

Text #18

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7168. Min Act: -0.0000

Data Index: 262472 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 1.1827. Min Range: -1.1827

Max Act: 0.7026. Min Act: -0.0000

Data Index: 832381 (C4 (Web Text))

Max Activating Token Index: 566