Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 165 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.1480. Min Act: -0.0000

Data Index: 482410 (C4 (Web Text))

Max Activating Token Index: 552

Text #1

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.1651. Min Act: -0.0000

Data Index: 1340040 (C4 (Web Text))

Max Activating Token Index: 315

Text #2

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.1081. Min Act: -0.0000

Data Index: 1192328 (C4 (Web Text))

Max Activating Token Index: 456

Text #3

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.1152. Min Act: -0.0000

Data Index: 67631 (C4 (Web Text))

Max Activating Token Index: 676

Text #4

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.0511. Min Act: -0.0000

Data Index: 345747 (C4 (Web Text))

Max Activating Token Index: 615

Text #5

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.0404. Min Act: -0.0000

Data Index: 812664 (C4 (Web Text))

Max Activating Token Index: 735

Text #6

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.0786. Min Act: 0.0000

Data Index: 1088494 (C4 (Web Text))

Max Activating Token Index: 200

Text #7

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9170. Min Act: -0.0000

Data Index: 833905 (C4 (Web Text))

Max Activating Token Index: 646

Text #8

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9785. Min Act: -0.0000

Data Index: 528363 (C4 (Web Text))

Max Activating Token Index: 724

Text #9

Max Range: 1.1651. Min Range: -1.1651

Max Act: 1.0098. Min Act: -0.0000

Data Index: 554959 (C4 (Web Text))

Max Activating Token Index: 672

Text #10

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9119. Min Act: -0.0000

Data Index: 208453 (C4 (Web Text))

Max Activating Token Index: 241

Text #11

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9065. Min Act: -0.0000

Data Index: 1057599 (C4 (Web Text))

Max Activating Token Index: 943

Text #12

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8963. Min Act: 0.0000

Data Index: 329739 (C4 (Web Text))

Max Activating Token Index: 388

Text #13

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8871. Min Act: 0.0000

Data Index: 1312216 (C4 (Web Text))

Max Activating Token Index: 762

Text #14

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9203. Min Act: 0.0000

Data Index: 1242759 (C4 (Web Text))

Max Activating Token Index: 445

Text #15

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.9107. Min Act: -0.0000

Data Index: 76899 (C4 (Web Text))

Max Activating Token Index: 193

Text #16

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8530. Min Act: -0.0000

Data Index: 745099 (C4 (Web Text))

Max Activating Token Index: 228

Text #17

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8503. Min Act: -0.0000

Data Index: 905585 (C4 (Web Text))

Max Activating Token Index: 776

Text #18

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8210. Min Act: -0.0000

Data Index: 1242920 (C4 (Web Text))

Max Activating Token Index: 815

Text #19

Max Range: 1.1651. Min Range: -1.1651

Max Act: 0.8095. Min Act: -0.0000

Data Index: 47743 (C4 (Web Text))

Max Activating Token Index: 177